Table of Contents

Search

  1. Preface
  2. Introduction to Mass Ingestion
  3. Prepare
  4. Create
  5. Deploy
  6. Run
  7. Monitor
  8. Appendix A: infacmd mi Command Reference

Mass Ingestion Guide

Mass Ingestion Guide

Run Overview

Run Overview

After you deploy a mass ingestion specification, you can run the specification to ingest the data.
When you run the specification, you can select one of the following load types:
  • Full load. A full load ingests all of the data to the target. When you use a full load, the Spark engine deletes the existing data in the Hive or HDFS target and replaces the data with the data that is configured in the specification.
  • Incremental load. An incremental load ingests only the incremental data to the target. The Spark engine appends the incremental data to the target or overwrites the target with the incremental data depending on the mode that you configure in the specification.
Each time that you run the specification, the Mass Ingestion Service generates a new run instance for the specification. The Data Integration Service processes the instance and connects to the Hadoop environment. In the Hadoop environment, the Spark engine runs the individual ingestion mappings jobs that ingest the data in the source tables to the target.
If the ingestion job stops responding or takes an excessive amount of time to complete, you can cancel the job in the Monitor tool.
For information on canceling ingestion jobs, see Canceling Ingestion Jobs in the "Monitor" chapter.

0 COMMENTS

We’d like to hear from you!