Taskflows

Back Next

Ingestion Task step

Use an Ingestion Task step to leverage a file ingestion task for taskflow orchestration. In the Ingestion Task step, you can select an existing file ingestion task.

File ingestion tasks that use a file listener component as the source are not available for selection.

You might want to perform data integration operations after moving files to an intermediate location and before transferring the files to the target. In this scenario, you can use the Ingestion Task step in conjunction with the Data Task step.

For example, you can use the Ingestion Task step to read a large number of files from a source location and write them to an intermediate location. You can then use the Data Task step to perform data integration operations on the files and use another Ingestion Task step to write the updated files to the final target location.

The following sections describe the Ingestion Task step properties:

General properties

In the general properties, you can specify a descriptive name for the Ingestion Task step.

The name can contain only alphanumeric characters, underscores (_), spaces, and Unicode characters. The name can't contain curly brackets {}.

Ingestion Task step properties

In the Ingestion Task step properties, you can select the file ingestion task that you want to add to the taskflow. When you select a file ingestion task, you see a description and output fields for the file ingestion task.

The

Output Fields

section shows the following fields:

Property	Description
Success Files	Number of files successfully transferred to the target.
Error Files	Number of files not transferred to the target.
fileDetails	A list of objects that return the file transfer details such as a list of created, updated, and deleted files along with the file count. The fileDetails field displays a maximum of 1,500 records even if the total number of processed files exceeds 1,500.

Input fields

You can add input fields to override the following file ingestion task properties:

General properties:

Maximum File Limit

. The maximum number of records to display in the

fileDetails

field in the

Output Fields

section. When you select a file ingestion task in the Ingestion Task step, the

Maximum File Limit

field is added by default. The maximum number you can specify is 1500. Default is

Runtime Environment

. Runtime environment that contains the Secure Agent used to run the task.

Source Connection

. The connection that the file ingestion task uses to read from the source.

Target Connection

. The connection that the file ingestion task uses to write to the target.

Source properties:

Batch Size

. The number of files that a file ingestion task can transfer in a batch.

File Pattern

. The file name pattern to use for selecting the files to transfer.

Folder Path

. The folder path from where the files are transferred. The default value is the folder path specified in the connection.

Source Directory

. The directory from where the files are transferred. The default value is the source directory specified in the connection.

Target properties:

Folder Path

. The folder path to which the files are transferred. The default value is the folder path specified in the connection.

Target Directory

. The directory to which the files are transferred. The default value is the target directory specified in the connection.

Events

Configure the

Events

properties to add timers to an ingestion task.

Use a Timer event to perform an action based on a schedule. The action could be either at a specific time or after an interval.

When you add a timer to an Ingestion Task step, a new branch appears. Add an event to this branch and specify whether you want the event to run

a specific time or

After

an interval.

In the following image, the event on the timer branch is a Decision step (Decision 1) that occurs five minutes after the main ingestion task (Ingestion Task 1) starts:

The image shows a Decision step that occurs five minutes after the main ingestion task starts.

Main branch

Timer branch

In the example, the timer branch runs five minutes after Ingestion Task 1 starts. If Ingestion Task 1 finishes before Decision 1, the timer branch is not executed.

You can select the

Interrupting

option if you want the timer to interrupt the main ingestion task.

The following image shows an interrupting timer set to occur five minutes after the main ingestion task starts:

The image shows an interrupting timer set to occur five minutes after the main ingestion task starts.

Main branch

Timer branch

In the example, Ingestion Task 2 executes after five minutes and interrupts Ingestion Task 1. The taskflow executes only Ingestion Task 2 and then ends. Ingestion Task 1 has no output in this taskflow instance. You see no output fields for Ingestion Task 1 in the job details for the taskflow.

If Ingestion Task 1 completes before the timer, the taskflow executes only Ingestion Task 1 and ends.

If you delete the End step on the timer branch of an interrupting timer, the timer branch rejoins the main branch.

The following image shows an interrupting timer branch with the End step deleted:

Main branch

Timer branch

In the example, Ingestion Task 2 executes after five minutes and interrupts Ingestion Task 1. The timer branch rejoins the main branch. The taskflow executes Ingestion Task 2, a Decision step, a Parallel Paths step, and then ends.

If Ingestion Task 1 completes before the timer, the taskflow executes Ingestion Task 1, a Decision step, a Parallel Paths step, and then ends.

Error handling properties

Use the

Error Handling

tab to indicate how you want the taskflow to behave when an Ingestion Task step encounters an error. After you select a file ingestion task, you can configure the

On Error

property.

The

On Error

property defines the path that a taskflow takes when it encounters an error in an Ingestion Task step. An error occurs when an Ingestion Task step fails. You can choose from the following options:

Select

Ignore

to ignore the error and move to the next step.

Select

Suspend Taskflow

to move the taskflow to the suspended state when it encounters an error. You can resume the taskflow instance from the

All Jobs

Running Jobs

, or

My Jobs

page. The taskflow resumes from the step at which it was suspended. If you know the reason for the error, correct the issue and then resume the taskflow.

Select

Custom error handling

to handle the error in a manner you choose. If you select

Custom error handling

, two branches appear. The first branch is the path that the taskflow follows if no error occurs. The second branch is the custom path that the taskflow follows if an error occurs.

The following image shows a custom error handling path with an Ingestion Task step: