Tasks

Back Next

Incremental loading

You can configure a

data loader

task to load data to the target incrementally. When the task loads data incrementally, only new and changed data is loaded to the target each time you re-run the task. Incremental loading increases task performance since fewer rows are loaded.

To configure the task to load data incrementally, select

Yes

under

Load to existing tables

on the

Connect Target

page.

When you load data incrementally, the task runs most efficiently when you configure the following options in the source:

Primary key fields are detected automatically or entered manually.: The task uses the primary key fields to identify rows when updating or inserting data into the target tables. If there are no primary key fields, the task can insert data into the target tables, but it can't update existing rows. This can create duplicate rows in the target tables.
Watermark fields are detected automatically or entered manually.: To load data incrementally, the source objects must contain watermark fields. The task uses the watermark fields to determine which records have been added or changed since the last task run. If there are no watermark fields, the task can't determine which records are new or changed. So, it processes all records in the source objects each time the task runs.

If the source object structure changes between task runs, the

data loader

task detects the changes and alters the target tables to match the source objects. Therefore, if you add a field to a source object or change a field's data type, the task adds the new column or updates the column data type in the corresponding target table. The task does not delete any existing target fields, however. If you want to delete unnecessary target fields, you can either delete the fields manually or configure the task to drop and re-create the target tables.

If the name or data type of the watermark column changes between task runs, or if you change the watermark column between task runs, the task performs a full load even if you've configured the target to load to existing tables.

Rename Saved Search

Table of Contents

Tasks

Tasks

Incremental loading

Incremental loading