Table of Contents

Search

  1. Preface
  2. Introduction to Mass Ingestion
  3. Prepare
  4. Create
  5. Deploy
  6. Run
  7. Monitor
  8. Appendix A: infacmd mi Command Reference

Mass Ingestion Guide

Mass Ingestion Guide

Incremental Load

Incremental Load

Use an incremental load to ingest only incremental data to the target. Before you can use an incremental load, you must enable incremental load and configure incremental load options in the mass ingestion specification.
You might want to use an incremental load since a full load can be time- and resource-intensive, especially if the data is largely unchanged between runs of a specification. For a more cost-efficient solution, you can use an incremental load to ingest only the incremental data each time that the specification runs.
Before you begin running incremental loads on a specification, run an initial full load. An initial full load allows the Spark engine to internally persist the original source data and use the persisted values as a basis to fetch incremental data in an incremental load. After the Spark engine persists the original source data, you can ingest incremental data based on the original source data to any target.
If you run an incremental load using append mode, the target must contain existing tables or files so that the Spark engine can append incremental data to the existing data. To create the tables or files, run a full load.

0 COMMENTS

We’d like to hear from you!