Amazon S3 V2 Connector

Back Next

Incrementally loading files

You can incrementally load source files in a directory to read and process only the files that have changed since the last time the mapping task ran.

You can incrementally load files only from mappings in advanced mode. Ensure that all of the source files exist in the same Cloud environment.

To incrementally load source files, select

Incremental File Load

and Directory
as the source type in the advanced read options of the Amazon S3 V2 data object.

When a mapping incrementally loads files, it reads files from a source directory that have changed between the time of the last mapping run and the current time minus the buffer time. The default buffer time for Amazon S3 V2 connection is 1200 seconds (20 minutes).

For example, if you run a job at 2:00 p.m. with a buffer time of 1200 seconds (20 minutes), the job loads files changed before 1:40 p.m. The 20-minute buffer ensures that the job loads only complete files, since uploading objects on Amazon S3 can take a few minutes to complete.

When you configure a mapping task, the Incremental File Load
section lists the Source transformations that incrementally loads files and the time that the last job completed loading the files. By default, the next job that runs checks for files modified after the last load time.

The image shows the details of incremental file load

You can also override the load time that the mapping uses to look for changed files in the specified source directory. You can reset the incremental file load settings to perform a full load of all the changed files in the directory, or you can configure a time that the mapping uses to look for changed files.

A mapping in advanced mode that incrementally load a directory that contains complex file formats such as Parquet and Avro fails if there are no new or changed files in the source since the last run.

For more information on incremental loading, see Reprocessing incrementally-loaded source files in

Tasks

in the Data Integration documentation.

Configure the buffer time

You can shorten the buffer time to reduce latency in file detection. You can lengthen the buffer time if you encounter issues with loading incomplete files. To avoid data loss due to minor time differences between Informatica services and your cloud storage system, keep the buffer time set to at least 300 seconds (5 minutes).

To configure the buffer time, set the

Spark.FileIncrementalLoadCorrectionTimeWindow=<buffer time>

in the advanced session property in the mapping task.

Amazon S3 V2 sources

Download Guide

Watch

Comments

Cloud Connectivity Homepage

Communities

Knowledge Base

Success Portal

0 COMMENTS

We’d like to hear from you! Log in to comment.

Alessio Giordani - November 08, 2024

Hello Documentation team,

I'm experiencing that on S3 the Incremental File Load time buffer is actually 20 minutes, meaning that when IFL is enabled, the mapping process S3 files older than 20 minutes. Can you please update the user guide?

Thanks,

Alessio

Informatica Documentation Team - November 13, 2024

Hello Alessio,

Thanks for your comment. We're glad to let you know that we addressed your comment on this page. We have updated the Incremental File Load time buffer to 20 minutes.

Rename Saved Search

Table of Contents

Amazon S3 V2 Connector

Amazon S3 V2 Connector

Incrementally loading files

Incrementally loading files

Configure the buffer time