Administrator
- Administrator
- All Products
Field | Type | Description | Supported target types |
|---|---|---|---|
connection | String | Name of the connection for the target system. | All supported targets |
schema | String | The target schema where Data Ingestion and Replication creates the target
objects or tables. | Microsoft Azure Synapse Analytics, Snowflake Data Cloud, Amazon Redshift,
Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL |
bucket | String | The name of an existing bucket container that stores, organizes, and controls
access to the data objects that you load to the target. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage |
directory | String | Name of a root directory to use for storing output files for an incremental
load task. If you select the Connection Directory as Parent option, you can still
optionally specify a task target directory. It will be appended to the parent
directory to form the root for the data, schema, cycle completion, and cycle
contents directories. This field is required if the {TaskTargetDirectory}
placeholder is specified in patterns for any of the following directory
fields. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud
Storage |
snowflakeRole | String | The Snowflake role assigned to the data ingestion and replication user to load
data to Snowflake. | Snowflake Data Cloud |
useTableNameAsTopicName | Boolean | Indicates whether the task writes messages that contain source data to separate
topics, one for each source table or object, or writes all messages to a single
topic. Options are:
| Kafka |
includeSchemaName | Boolean | When you set useTableNameAsTopicName to true for a Kafka target, this setting
adds the source schema name in the table-specific topic names. The topic names then
have the format schemaname_tablename. Options are:
| Kafka |
tablePrefix | String | When you set useTableNameAsTopicName to true for a Kafka target, this parameter
specifies an optional prefix to add to the table-specific topic names. For
example, if you specify myprefix_, the topic names have the format
myprefix_tablename. If you omit the underscore (_) after the prefix, the prefix is
prepended to the table name. | Kafka |
tableSuffix | String | When you set useTableNameAsTopicName to true for a Kafka target, this parameter
specifies an optional suffix to add to the table-specific topic names. For
example, if you specify _mysuffix, the topic names have the format
tablename_mysuffix. If you omit the underscore (_) before the suffix, the suffix
is appended to the table name. | Kafka |
topicName | String | The name of the single Kafka topic to which all messages that contain source
data will be written. Use this parameter if useTableNameAsTopicName is set to
false. | Kafka |
stage | String | The name of the internal staging area that holds the data read from the source
before the data is written to the target tables. This name must not include spaces.
If the staging area does not exist, it will be automatically created. | Snowflake Data Cloud |
auditData | String | For incremental load and combined initial and incremental load jobs, indicates
how source DML changes, including inserts, updates, and deletes, are applied to the
target. Options are:
| |
outputFormat | String | The format of the output file. Options are:
Output
files in CSV format use double-quotation marks ("") as the delimiter for each
field. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google
Cloud Storage |
parquetFormat | Boolean | If AVRO is specified as the output format in the outputFormat parameter for the
target, set this parameter to true to write data in uncompressed Parquet format.
Alternatively, you can set the outputFormat parameter to PARQUET and not include
this parameter. Options are:
If you set this option
to true, you must install Visual C++ Redistributable Packages for Visual Studio
2013 on the computer where the Secure Agent runs. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google
Cloud Storage |
avroFormat | String | If AVRO is specified as the output format, specifies the format of the Avro
schema that will be created for each source table or object. Options are:
| Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google
Cloud Storage |
avroSerializationFormat | String | If AVRO is specified as the output format, specify the serialization format of
the Avro output files. Options are:
| Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google
Cloud Storage |
avroSchemaDirectory | String | If AVRO is specified as the output format, specifies the local directory where
Data Ingestion and Replication stores Avro schema definitions for each source table
or object. Schema definition files have the following naming pattern:
schemaname_tablename.txt If
you do not specify this directory, no Avro schema definition file is
produced. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google
Cloud Storage |
fixedDirectoryForEachTable | Boolean | For an initial load task, indicates whether to use the source table or object
names as the names of the directories to which the task writes flat files that
contain source data for all job runs. Options are :
| Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud
Storage |
fileCompressionType | String | The file compression type to optionally use for AVRO or CSV output files.
Options are:
| Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud
Storage |
avroCompressionType | String | The optional Avro compression type if the output format is set to AVRO.
Options are:
None , which means no compression is used. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud
Storage |
parquetCompressionType | String | The optional Parquet compression type to use when the output format is set to
PARQUET or AVRO by the parquetFormat true parameter. Options are:
| Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud
Storage |
deflateCompressionLevel | Integer | The compression level of 0-9 to use for Deflate compression type, which is set
by the avroCompressionType parameter. Default is 0. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud
Storage |
addDirectoryTags | Boolean | For incremental load and combined initial and incremental load tasks, indicates
whether the task adds the "dt=" prefix to the names of apply cycle directories to be
compatible with the naming convention for Hive partitioning. Options are:
| Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage |
directoryTags | String | This field is dependent on the addDirectoryTags field. For example:
addDirectoryTags: true, directoryTags:
\directory. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage |
renamingRules | List | Optional rules for renaming target tables that correspond to the source tables
or objects. For example:
| Amazon S3, Microsoft Azure Data Lake Storage Gen2, Microsoft Azure Synapse
Analytics, Flat File, Snowflake Data Cloud, Amazon Redshift, Google BigQuery,
Databricks, Oracle, Microsoft SQL Server, PostgreSQL, Google Cloud Storage |
dataTypeRules | List | Optional data type mapping rules that override the default source-to-target
data-type mapping rules. The default mappings are described under Default Data-Type
Mappings in the Application Ingestion and Replication and
Database Ingestion and Replication documentation.For example: - source: Int target: String | Microsoft Azure Synapse Analytics, Snowflake Data Cloud, Amazon Redshift,
Google BigQuery, Databricks |
cdcCompatibleFormat | Boolean | Indicates whether to include UNDO data in the output. Default is
false. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google
Cloud Storage |
useGenericDatatypeFramework | Boolean | Set this parameter to true to use the generic data type framework for mapping
source data types to target data types. | All supported targets |
Field | Type | Description | Supported target types |
|---|---|---|---|
createUnmanagedTables | Boolean | Set this parameter to true if you want the task to create Databricks target
tables as unmanaged tables. After you deploy the task, you cannot change this
parameter to switch to managed tables. Default is false, which creates managed
tables. | Databricks |
unmanagedTablesParentDirectory | String | If you create Databricks unmanaged tables, specify a parent directory in Amazon
S3 or Microsoft Azure Data Lake Storage to hold the Parquet files that are generated
for each target table when captured DML records are processed. For example:
/data/unmanaged | Databricks |
writerTargetTableFormat | String | The format of the target tables, such as standard or Iceberg. To enable the
use of Snowflake-managed Iceberg tables, set this property to
iceberg . | Snowflake Data Cloud |
writerTargetTableExternalVolumeName | String | If the target uses the Iceberg table format, the name of the external volume
used for the target tables. | Snowflake Data Cloud |
writerTargetTableExternalStorageBaseLocation | String | If the target uses the Iceberg table format, the base URL for the external
storage location. If you do not specify this value, the base URL from the external
volume definition is used. | Snowflake Data Cloud |
writerSnowflakeIngestionMethod | String | Set to SNOWPIPE_STREAMING to use the Snowpipe Streaming API to quickly stream
rows of data directly to Snowflake Data Cloud target tables with low latency instead
of first writing the data to stage files. This option is applicable for all load
types. Ensure that the Snowflake connection uses KeyPair
authentication. | Snowflake Data Cloud |
writerSnowflakeDeferredMergeInterval | Integer | If you set writerSnowflakeIngestionMethod to SNOWPIPE_STREAMING, you can
optionally set the frequency, in seconds, at which change data rows are merged and
applied to the Snowflake target tables. The merge frequency affects how often the
stream change data is merged to the Snowflake base table. A Snowflake view joins
the stream change data with the base table. Set this value to balance the costs of
merging data to the base table with the performance of view join
processing. This field applies to incremental load and combined initial and
incremental load tasks. Valid values are 60 through 604800 seconds. Default
is 3600 seconds. | Snowflake Data Cloud |
addHeadersToCSVFile | Boolean | If you specify CSV as the output format, indicates whether to add a
header row with column names to the CSV output files. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud
Storage |
addOperationType | Boolean | For targets in an incremental load or combined initial and incremental load
task, set this parameter to true to add a metadata column that includes the source
SQL operation type in the output that the job replicates to the target. For an
incremental load or combined initial and incremental load task, default is true.
For an initial load task, default is false. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google
Cloud Storage |
addOperationTime | Boolean | For targets in an incremental load or combined initial and incremental load
task, set this parameter to true to add a metadata column that includes the source
SQL operation time in the output that the job replicates to the target. For initial
loads, the job always writes the current date and time. Default is false for all
load types. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google
Cloud Storage |
addOperationOwner | Boolean | For targets in an incremental load or combined initial and incremental load
task, set this parameter to true to add a metadata column that includes the owner of
the source SQL operation in the output that the job replicates to the target. For
initial loads, the job always writes "INFA" as the owner. Default is false.
| Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google
Cloud Storage |
addOperationTransactionId | Boolean | For targets in an incremental load or combined initial and incremental load
task, set this parameter to true to add a metadata column that includes the source
transaction ID in the output that the job replicates to the target for SQL
operations. Default is false. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google
Cloud Storage |
addBeforeImages | Boolean | For targets in an incremental load or combined load task, set this parameter to
true to include UNDO data in the output that the job writes to the target. Default
is false. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google
Cloud Storage |
asyncWrite | Boolean | Controls whether to use synchronous delivery of messages to Kafka. Options
are:
| Kafka |
producerConfigurationProperties | String | A comma-separated list of key=value pairs to enter Kafka producer properties
for Apache Kafka, Confluent Kafka, or Kafka-enabled Event Hubs targets. If you
have a Confluent target that uses Confluent Schema Registry to store schemas, you
must specify the following properties: schema.registry.url=url,
key.serializer= org.apache.kafka.common.serialization.StringSerializer,
value.serializer= io.confluent.kafka.serializers.KafkaAvroSerializer | Kafka |
writerHelperThreadCount | Integer | The number of writer helper threads that are used to convert incoming change
data rows or initial unload rows to the output format configured for the target,
such as Avro, CSV, or Parquet. Default value is 2. | All supported targets |
writerDistributorsCount | Integer | The number of distributors that can run on separate threads in parallel to
process data during an initial load job or the unload phase of a combined load job
when the Writer Unload Multiple Distributors custom property is set to true. Using
parallel distributor threads can improve job performance, particularly for
high-volume data transfers. Default value is 3. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Snowflake Data Cloud, Google
BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL, Google Cloud
Storage |
writerUnloadUseMultipleDistributors | Boolean | Indicates whether multiple distributor threads can be used to process data in
parallel during initial load jobs and the unload phase of combined load jobs.
Default value is false. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Snowflake Data Cloud, Google
BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL, Google Cloud
Storage |
auditColumnsPrefix | String | Adds a prefix to the names of the added metadata columns to easily identify
them and to prevent conflicts with the names of existing columns. The default value
is INFA_. | Microsoft Azure Synapse Analytics, Snowflake Data Cloud, Amazon Redshift,
Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL |
jsonFormatType | String | The format type used when writing output data in JSON format. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google
Cloud Storage |
checkpointsInTarget | Boolean | Set to true if you want the job to save a checkpoint in the Kafka header each
time a source transaction commits. When the job is restarted, checkpoint information
is retrieved from the Kafka header. | Kafka |
addLastReplicatedTime | Boolean | Adds a metadata column that records the timestamp in UTC format at which a
record was inserted or last updated in the target table. For initial loads, all
loaded records have the same timestamp. For incremental and combined initial and
incremental loads, the column records the timestamp of the last DML operation that
was applied to the target. | Databricks, Amazon Redshift, Snowflake Data Cloud |
caseTransformation | String | Controls the case of letters in the target names. Case transformation
strategy options to handle the case of letters in generated target table (or
object) names and column (or field) names are:
| Google BigQuery, Snowflake Data Cloud |
useConnectionDir | Boolean | Uses the parent directory specified in the connection properties. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage |
addOperationSequence | Boolean | Adds a metadata column that records a generated, ascending sequence number for
each change operation that the job inserts into the target tables. The sequence
number reflects the change stream position of the operation. This field is
applicable when you use Apply Mode. | Microsoft Azure Synapse Analytics, Snowflake Data Cloud, Amazon Redshift,
Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL |
cycleId | Boolean | Set to true to add a metadata column that includes the cycle ID of each CDC
cycle in each target table. A cycle ID is a number that's generated by the CDC
engine for each successful CDC cycle. If you integrate the job with Data Integration
taskflows, the job can pass the minimum and maximum cycle IDs in output fields to
the taskflow so that the taskflow can determine the range of cycles that contain new
CDC data. This capability is useful if data from multiple cycles accumulates before
the previous taskflow run completes. | Microsoft Azure Synapse Analytics, Snowflake Data Cloud, Amazon Redshift,
Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL |
schemaDirectory | String | A custom directory for storing the schema file, if you want to store it in a
directory other than the default. This property is optional. For initial loads,
the schema is stored in the data directory by default. For incremental load and
combined initial and incremental load tasks, the default value is
{TaskTargetDirectory}/data/{TableName}/schema .You
can use the same placeholders as for the data directory. Enclose placeholders in
curly brackets { }. If you include the toUpper or toLower function, put the
placeholder name in parentheses and enclose both the function and placeholder in
curly brackets, for example: {toLower(SchemaName)}. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage |
cdcDataDirectory | String | For incremental load and combined initial and incremental load tasks, the path
to the subdirectory that contains the cdc-data data files. You can build the path
from a folder path, timestamp values, schema name, and table name. Enclose each
placeholder in curly brackets { }. Placeholder values are not case sensitive. Use
{TaskTargetDirectory} for a task-specific base directory on the target instead of
the folder path specified in the connection properties. Default value is
{TaskTargetDirectory}/data/{TableName}/data .Placeholders:
{TaskTargetDirectory}, {Timestamp}, {yy}, {yyyy}, {mm}, {dd}, {SchemaName},
{toUpper(SchemaName)}, {toLower(SchemaName)}, {TableName}, {toUpper(TableName)},
{toLower(TableName)}. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage |
cdcCompletionDirectory | Boolean | For incremental load and combined initial and incremental load tasks, the
path to the directory that contains the cycle completed file. {TaskTargetDirectory}/cycle/completed . | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage |
cdcContentsDirectory | String | For incremental load and combined initial and incremental load tasks, the
path to the directory that contains the cycle contents files. TaskTargetDirectory}/cycle/contents . | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage |
cdcDataDirectoryPartitioning | Boolean | For incremental load and combined initial and incremental load tasks, indicates
whether to create a timestamp subdirectory for each CDC cycle under each data
directory. Options are:
| Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage |
cdcSummaryDirectoryPartitioning | Boolean | For incremental load and combined initial and incremental load tasks, indicates
whether to create a timestamp subdirectory for each CDC cycle under the summary
contents and completed subdirectories. Options are:
| Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage |
cdcContentsListFiles | Boolean | For incremental load and combined initial and incremental load tasks, indicates
whether to list individual data files under the contents subdirectory. Options
are:
If you set cdcSummaryDirectoryPartitioning to false, the default is
true. All individual files are listed in the contents subdirectory unless you
configure custom subdirectories by using placeholders, such as for timestamp or
date. If you set cdcDataDirectoryPartitioning to true, you can still set
this property to true to list individual files and group them by CDC
cycle. | Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage |
Field | Type | Description | Supported target types |
|---|---|---|---|
forceRunOnOtherAgent | Boolean | Set to true to force the job to run on a Secure Agent other than the one it
would run on by default. | All supported targets |
forceRunOnSpecificAgent | String | Set this property to force an application or database ingestion and replication
job to run on a specific agent in the group. This setting applies only to
incremental load jobs and the incremental CDC portion of combined load jobs. This capability is helpful when you want to manually assign CDC jobs to a
specific agent specified by the agent ID. For example, "forceRunOnSpecificAgent": "agent-007" To find the ID, go to Runtime Environments in Administrator, expand the runtime
environment for the Secure Agent group, click the agent, and copy the agent ID
from the browser URL. | All applicable targets |