Tasks

Back Next

Reprocessing incrementally-loaded source files example

You have a source directory that receives a new data file every six hours. In Data Integration, you have a mapping that incrementally loads these files to a target and a mapping task that is scheduled to run the mapping at 3:00 a.m. every day.

The following image shows the contents of the source and target directories over two days:

A timeline spanning from February 3 through February 4 has four highlighted points, which are expanded to list the contents of the source and target directories at each point. At 1 A.M. on February 3, the target directory has four files numbered 1 through 4 and the source directory has four files numbered 5 through 8. At 3:15 A.M. that day, after the job runs, the source has no files and the target has eight files numbered 1 through 8. At 1 A.M. the next day, the source has four files numbered 9 through 12 and the target has the same eight files as before. At 3:15 A.M. that day, after the job runs, the source has no files and the target has all 12 files.

Mid-day on February 4, you realize that incorrect data was loaded the previous day, February 3, so your target contains incorrect data for the files created on February 2. After fixing the files in the source directory, you run a job with advanced options to reprocess the incorrect files.

In the advanced options, you choose to read incrementally-loaded source files that were modified between 02/02/2022 00:00:01 and 02/03/2022 00:00:00. You click

Run

to run the mapping task with the advanced options. The reprocessing job processes the files modified within the selected time interval and writes them to the target.

When the next scheduled job runs at 3 a.m. on February 5, it incrementally loads the files created on February 4, as usual.

Reprocessing incrementally-loaded source files

Download Guide

Watch

Comments

Cloud Data Integration Homepage