Hi, I'm Ask INFA!
What would you like to know?
ASK INFAPreview
Please to access Ask INFA.

Table of Contents

Search

  1. Preface
  2. IDMC REST API
  3. REST API resource quick references
  4. Platform REST API version 2 resources
  5. Platform REST API version 3 resources
  6. Data Integration REST API
  7. Cloud Data Integration for PowerCenter (CDI-PC) REST API
  8. Data Ingestion and Replication REST API
  9. Customer Managed REST API
  10. RunAJob utility
  11. ParamSetCli utility
  12. REST API codes

REST API Reference

REST API Reference

Target attributes

Target attributes

The following table describes the target connection and task attributes in the request:
Field
Type
Description
Supported target types
connection
String
Name of the connection for the target system.
All supported targets
schema
String
The target schema where Data Ingestion and Replication creates the target objects or tables.
Microsoft Azure Synapse Analytics, Snowflake Data Cloud, Amazon Redshift, Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL
bucket
String
The name of an existing bucket container that stores, organizes, and controls access to the data objects that you load to the target.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
directory
String
Name of a root directory to use for storing output files for an incremental load task.
If you select the Connection Directory as Parent option, you can still optionally specify a task target directory. It will be appended to the parent directory to form the root for the data, schema, cycle completion, and cycle contents directories.
This field is required if the {TaskTargetDirectory} placeholder is specified in patterns for any of the following directory fields.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
snowflakeRole
String
The Snowflake role assigned to the data ingestion and replication user to load data to Snowflake.
Snowflake Data Cloud
useTableNameAsTopicName
Boolean
Indicates whether the task writes messages that contain source data to separate topics, one for each source table or object, or writes all messages to a single topic.
Options are:
  • true
    . Write messages to separate table-specific topics.
  • false
    . Write all messages to the single topic that has the name specified in the topicName field.
Kafka
includeSchemaName
Boolean
When you set useTableNameAsTopicName to true for a Kafka target, this setting adds the source schema name in the table-specific topic names. The topic names then have the format schemaname_tablename.
Options are:
  • true
    . Add the source schema name in the table-specific topic names.
  • false
    . Do not add the source schema name in the table-specific topic names.
Kafka
tablePrefix
String
When you set useTableNameAsTopicName to true for a Kafka target, this parameter specifies an optional prefix to add to the table-specific topic names.
For example, if you specify myprefix_, the topic names have the format myprefix_tablename. If you omit the underscore (_) after the prefix, the prefix is prepended to the table name.
Kafka
tableSuffix
String
When you set useTableNameAsTopicName to true for a Kafka target, this parameter specifies an optional suffix to add to the table-specific topic names.
For example, if you specify _mysuffix, the topic names have the format tablename_mysuffix. If you omit the underscore (_) before the suffix, the suffix is appended to the table name.
Kafka
topicName
String
The name of the single Kafka topic to which all messages that contain source data will be written. Use this parameter if useTableNameAsTopicName is set to false.
Kafka
stage
String
The name of the internal staging area that holds the data read from the source before the data is written to the target tables. This name must not include spaces. If the staging area does not exist, it will be automatically created.
Snowflake Data Cloud
auditData
String
For incremental load and combined initial and incremental load jobs, indicates how source DML changes, including inserts, updates, and deletes, are applied to the target.
Options are:
  • softDeletes. Apply source delete operations to the target as soft deletes. A soft delete marks the deleted row as deleted without actually removing it from the database. For example, a delete on the source results in a change record on the target with "D" displayed in the INFA_OPERATION_TYPE column.
    After enabling Soft Deletes, any update in the source table during normal or backlog mode results in the deletion of the matching record, insertion of the updated record, and marking of the INFA_OPERATION_TYPE operation as NULL in the target table. Similarly, inserting a record in the source table during backlog mode results in marking the INFA_OPERATION_TYPE operation as E in the target table record.
    Consider using soft deletes if you have a long-running business process that needs the soft-deleted data to finish processing, to restore data after an accidental delete operation, or to track deleted values for audit purposes.
  • audit. Apply an audit trail of every DML operation made on the source tables to the target. A row for each DML change on a source table is written to the generated target table along with the audit columns you select. The audit columns contain metadata about the change, such as the DML operation type, time, owner, transaction ID, generated ascending sequence number, and before image. Consider using the Audit apply mode when you want to use the audit history to perform downstream computations or processing on the data before writing it to the target database or when you want to examine metadata about the captured changes.
    If you use
    Query-based
    CDC method, this property is not applicable.
  • Standard. If you do not specify the auditData property, the task applies standard by default. In standard mode, the task accumulates the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target. For example, if an update followed by a delete occurs on the source row, no row is applied to the target. If multiple updates occur on the same column or field, only the last update is applied to the target. If multiple updates occur on different columns or fields, the updates are merged into a single update record before being applied to the target.
outputFormat
String
The format of the output file.
Options are:
  • AVRO
  • CSV
  • PARQUET
Default is CSV.
Output files in CSV format use double-quotation marks ("") as the delimiter for each field.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
parquetFormat
Boolean
If AVRO is specified as the output format in the outputFormat parameter for the target, set this parameter to true to write data in uncompressed Parquet format. Alternatively, you can set the outputFormat parameter to PARQUET and not include this parameter.
Options are:
  • true
    . Write data in uncompressed Parquet format.
  • false
    . Write data in AVRO format.
If you set this option to true, you must install Visual C++ Redistributable Packages for Visual Studio 2013 on the computer where the Secure Agent runs.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
avroFormat
String
If AVRO is specified as the output format, specifies the format of the Avro schema that will be created for each source table or object.
Options are:
  • Avro-Flat
    . This Avro schema format lists all Avro fields in one record.
  • Avro-Generic
    . This Avro schema format lists all columns or fields from a source table or object in a single array of Avro fields.
  • Avro-Nested
    . This Avro schema format organizes each type of information in a separate record.
Default is Avro-Flat.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
avroSerializationFormat
String
If AVRO is specified as the output format, specify the serialization format of the Avro output files.
Options are:
  • Binary
  • JSON
Default value is Binary.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
avroSchemaDirectory
String
If AVRO is specified as the output format, specifies the local directory where Data Ingestion and Replication stores Avro schema definitions for each source table or object.
Schema definition files have the following naming pattern: schemaname_tablename.txt
If you do not specify this directory, no Avro schema definition file is produced.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
fixedDirectoryForEachTable
Boolean
For an initial load task, indicates whether to use the source table or object names as the names of the directories to which the task writes flat files that contain source data for all job runs.
Options are :
  • true
    . Use the source table or object names as the names of the directories for each job run.
  • false
    . Create a new set of directories for each job run by using the following naming pattern: tablename_timestamp.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
fileCompressionType
String
The file compression type to optionally use for AVRO or CSV output files.
Options are:
  • Deflate
  • gzip
  • Snappy
If you do not specify this parameter, the output files are not compressed.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
avroCompressionType
String
The optional Avro compression type if the output format is set to AVRO.
Options are:
  • None
  • Bzip2
  • Deflate
  • Snappy
Default value is
None
, which means no compression is used.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
parquetCompressionType
String
The optional Parquet compression type to use when the output format is set to PARQUET or AVRO by the parquetFormat true parameter.
Options are:
  • None
  • gzip
  • Snappy
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
deflateCompressionLevel
Integer
The compression level of 0-9 to use for Deflate compression type, which is set by the avroCompressionType parameter.
Default is 0.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
addDirectoryTags
Boolean
For incremental load and combined initial and incremental load tasks, indicates whether the task adds the "dt=" prefix to the names of apply cycle directories to be compatible with the naming convention for Hive partitioning.
Options are:
  • true
    . Add the "dt=" prefix to the names of apply cycle directories.
  • false
    . Do not change the names of apply cycle directories.
Default is false.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
directoryTags
String
This field is dependent on the addDirectoryTags field.
For example:
addDirectoryTags: true,
directoryTags: \directory.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
renamingRules
List
Optional rules for renaming target tables that correspond to the source tables or objects.
For example:
- source: "*" target: "*_1"
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Microsoft Azure Synapse Analytics, Flat File, Snowflake Data Cloud, Amazon Redshift, Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL, Google Cloud Storage
dataTypeRules
List
Optional data type mapping rules that override the default source-to-target data-type mapping rules. The default mappings are described under Default Data-Type Mappings in the
Application Ingestion and Replication
and
Database Ingestion and Replication
documentation.
For example:
- source: Int target: String
Microsoft Azure Synapse Analytics, Snowflake Data Cloud, Amazon Redshift, Google BigQuery, Databricks
cdcCompatibleFormat
Boolean
Indicates whether to include UNDO data in the output.
Default is false.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
useGenericDatatypeFramework
Boolean
Set this parameter to true to use the generic data type framework for mapping source data types to target data types.
All supported targets
The following table describes the target advanced attributes that you can set for a task:
Field
Type
Description
Supported target types
createUnmanagedTables
Boolean
Set this parameter to true if you want the task to create Databricks target tables as unmanaged tables. After you deploy the task, you cannot change this parameter to switch to managed tables. Default is false, which creates managed tables.
Databricks
unmanagedTablesParentDirectory
String
If you create Databricks unmanaged tables, specify a parent directory in Amazon S3 or Microsoft Azure Data Lake Storage to hold the Parquet files that are generated for each target table when captured DML records are processed.
For example:
/data/unmanaged
Databricks
writerTargetTableFormat
String
The format of the target tables, such as standard or Iceberg.
To enable the use of Snowflake-managed Iceberg tables, set this property to
iceberg
.
Snowflake Data Cloud
writerTargetTableExternalVolumeName
String
If the target uses the Iceberg table format, the name of the external volume used for the target tables.
Snowflake Data Cloud
writerTargetTableExternalStorageBaseLocation
String
If the target uses the Iceberg table format, the base URL for the external storage location. If you do not specify this value, the base URL from the external volume definition is used.
Snowflake Data Cloud
writerSnowflakeIngestionMethod
String
Set to SNOWPIPE_STREAMING to use the Snowpipe Streaming API to quickly stream rows of data directly to Snowflake Data Cloud target tables with low latency instead of first writing the data to stage files. This option is applicable for all load types.
Ensure that the Snowflake connection uses KeyPair authentication.
Snowflake Data Cloud
writerSnowflakeDeferredMergeInterval
Integer
If you set writerSnowflakeIngestionMethod to SNOWPIPE_STREAMING, you can optionally set the frequency, in seconds, at which change data rows are merged and applied to the Snowflake target tables.
The merge frequency affects how often the stream change data is merged to the Snowflake base table. A Snowflake view joins the stream change data with the base table. Set this value to balance the costs of merging data to the base table with the performance of view join processing.
This field applies to incremental load and combined initial and incremental load tasks.
Valid values are 60 through 604800 seconds. Default is 3600 seconds.
Snowflake Data Cloud
addHeadersToCSVFile
Boolean
If you specify
CSV
as the output format, indicates whether to add a header row with column names to the CSV output files.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
addOperationType
Boolean
For targets in an incremental load or combined initial and incremental load task, set this parameter to true to add a metadata column that includes the source SQL operation type in the output that the job replicates to the target.
For an incremental load or combined initial and incremental load task, default is true. For an initial load task, default is false.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
addOperationTime
Boolean
For targets in an incremental load or combined initial and incremental load task, set this parameter to true to add a metadata column that includes the source SQL operation time in the output that the job replicates to the target. For initial loads, the job always writes the current date and time. Default is false for all load types.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
addOperationOwner
Boolean
For targets in an incremental load or combined initial and incremental load task, set this parameter to true to add a metadata column that includes the owner of the source SQL operation in the output that the job replicates to the target.
For initial loads, the job always writes "INFA" as the owner. Default is false.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
addOperationTransactionId
Boolean
For targets in an incremental load or combined initial and incremental load task, set this parameter to true to add a metadata column that includes the source transaction ID in the output that the job replicates to the target for SQL operations.
Default is false.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
addBeforeImages
Boolean
For targets in an incremental load or combined load task, set this parameter to true to include UNDO data in the output that the job writes to the target.
Default is false.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
asyncWrite
Boolean
Controls whether to use synchronous delivery of messages to Kafka.
Options are:
  • true
    . Use asynchronous delivery. Data Ingestion and Replication sends messages as soon as possible, without regard for the order in which the changes were retrieved from the source.
  • false
    . Use synchronous delivery. Kafka must acknowledge each message as received before Data Ingestion and Replication sends the next message. In this mode, Kafka is unlikely to receive duplicate messages. However, performance might be slower.
Default is true.
Kafka
producerConfigurationProperties
String
A comma-separated list of key=value pairs to enter Kafka producer properties for Apache Kafka, Confluent Kafka, or Kafka-enabled Event Hubs targets.
If you have a Confluent target that uses Confluent Schema Registry to store schemas, you must specify the following properties:
schema.registry.url=url, key.serializer= org.apache.kafka.common.serialization.StringSerializer, value.serializer= io.confluent.kafka.serializers.KafkaAvroSerializer
Kafka
writerHelperThreadCount
Integer
The number of writer helper threads that are used to convert incoming change data rows or initial unload rows to the output format configured for the target, such as Avro, CSV, or Parquet. Default value is 2.
All supported targets
writerDistributorsCount
Integer
The number of distributors that can run on separate threads in parallel to process data during an initial load job or the unload phase of a combined load job when the Writer Unload Multiple Distributors custom property is set to true. Using parallel distributor threads can improve job performance, particularly for high-volume data transfers. Default value is 3.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Snowflake Data Cloud, Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL, Google Cloud Storage
writerUnloadUseMultipleDistributors
Boolean
Indicates whether multiple distributor threads can be used to process data in parallel during initial load jobs and the unload phase of combined load jobs. Default value is false.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Snowflake Data Cloud, Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL, Google Cloud Storage
auditColumnsPrefix
String
Adds a prefix to the names of the added metadata columns to easily identify them and to prevent conflicts with the names of existing columns. The default value is INFA_.
Microsoft Azure Synapse Analytics, Snowflake Data Cloud, Amazon Redshift, Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL
jsonFormatType
String
The format type used when writing output data in JSON format.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
checkpointsInTarget
Boolean
Set to true if you want the job to save a checkpoint in the Kafka header each time a source transaction commits. When the job is restarted, checkpoint information is retrieved from the Kafka header.
Kafka
addLastReplicatedTime
Boolean
Adds a metadata column that records the timestamp in UTC format at which a record was inserted or last updated in the target table. For initial loads, all loaded records have the same timestamp. For incremental and combined initial and incremental loads, the column records the timestamp of the last DML operation that was applied to the target.
Databricks, Amazon Redshift, Snowflake Data Cloud
caseTransformation
String
Controls the case of letters in the target names.
Case transformation strategy options to handle the case of letters in generated target table (or object) names and column (or field) names are:
  • Same as source
    . Use the same case as the source table (or object) names and column (or field) names. Enter value "none" to use this option.
  • UPPERCASE
    . Use all uppercase.
  • lowercase
    . Use all lowercase.
Google BigQuery, Snowflake Data Cloud
useConnectionDir
Boolean
Uses the parent directory specified in the connection properties.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
addOperationSequence
Boolean
Adds a metadata column that records a generated, ascending sequence number for each change operation that the job inserts into the target tables. The sequence number reflects the change stream position of the operation.
This field is applicable when you use Apply Mode.
Microsoft Azure Synapse Analytics, Snowflake Data Cloud, Amazon Redshift, Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL
cycleId
Boolean
Set to true to add a metadata column that includes the cycle ID of each CDC cycle in each target table. A cycle ID is a number that's generated by the CDC engine for each successful CDC cycle. If you integrate the job with Data Integration taskflows, the job can pass the minimum and maximum cycle IDs in output fields to the taskflow so that the taskflow can determine the range of cycles that contain new CDC data. This capability is useful if data from multiple cycles accumulates before the previous taskflow run completes.
Microsoft Azure Synapse Analytics, Snowflake Data Cloud, Amazon Redshift, Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL
schemaDirectory
String
A custom directory for storing the schema file, if you want to store it in a directory other than the default. This property is optional.
For initial loads, the schema is stored in the data directory by default. For incremental load and combined initial and incremental load tasks, the default value is
{TaskTargetDirectory}/data/{TableName}/schema
.
You can use the same placeholders as for the data directory. Enclose placeholders in curly brackets { }. If you include the toUpper or toLower function, put the placeholder name in parentheses and enclose both the function and placeholder in curly brackets, for example: {toLower(SchemaName)}.
The task writes the schema only to output data files in CSV format. Parquet and Avro data files contain their own embedded schema.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
cdcDataDirectory
String
For incremental load and combined initial and incremental load tasks, the path to the subdirectory that contains the cdc-data data files.
You can build the path from a folder path, timestamp values, schema name, and table name. Enclose each placeholder in curly brackets { }. Placeholder values are not case sensitive. Use {TaskTargetDirectory} for a task-specific base directory on the target instead of the folder path specified in the connection properties.
Default value is
{TaskTargetDirectory}/data/{TableName}/data
.
If you use Amazon S3 with the Apache Iceberg Open Table format, this property does not apply. Files are saved under the warehouse base directory.
Placeholders: {TaskTargetDirectory}, {Timestamp}, {yy}, {yyyy}, {mm}, {dd}, {SchemaName}, {toUpper(SchemaName)}, {toLower(SchemaName)}, {TableName}, {toUpper(TableName)}, {toLower(TableName)}.
Timestamp format is yyyymmdd_hhmissms.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
cdcCompletionDirectory
Boolean
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle completed file.
Default value is
{TaskTargetDirectory}/cycle/completed
.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
cdcContentsDirectory
String
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle contents files.
Default is {
TaskTargetDirectory}/cycle/contents
.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
cdcDataDirectoryPartitioning
Boolean
For incremental load and combined initial and incremental load tasks, indicates whether to create a timestamp subdirectory for each CDC cycle under each data directory.
Options are:
  • true. Create a timestamp subdirectory for each CDC cycle under the summary contents and completed subdirectories.
  • false. Write individual data files to the same directory without a timestamp, unless you define an alternative directory structure.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
cdcSummaryDirectoryPartitioning
Boolean
For incremental load and combined initial and incremental load tasks, indicates whether to create a timestamp subdirectory for each CDC cycle under the summary contents and completed subdirectories.
Options are:
  • true. Create a timestamp subdirectory for each CDC cycle under the summary contents and completed subdirectories.
  • false. Do not create timestamp subdirectories under the summary contents and completed subdirectories.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
cdcContentsListFiles
Boolean
For incremental load and combined initial and incremental load tasks, indicates whether to list individual data files under the contents subdirectory.
Options are:
  • true. List individual data files under the contents subdirectory.
  • false. Do not list individual data files under the contents subdirectory.
If you set cdcSummaryDirectoryPartitioning to false, the default is true. All individual files are listed in the contents subdirectory unless you configure custom subdirectories by using placeholders, such as for timestamp or date.
If you set cdcDataDirectoryPartitioning to true, you can still set this property to true to list individual files and group them by CDC cycle.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
The following table describes custom attributes that you can set for the target:
Field
Type
Description
Supported target types
forceRunOnOtherAgent
Boolean
Set to true to force the job to run on a Secure Agent other than the one it would run on by default.
All supported targets
forceRunOnSpecificAgent
String
Set this property to force an application or database ingestion and replication job to run on a specific agent in the group. This setting applies only to incremental load jobs and the incremental CDC portion of combined load jobs.
This capability is helpful when you want to manually assign CDC jobs to a specific agent specified by the agent ID.
For example, "forceRunOnSpecificAgent": "agent-007"
To find the ID, go to Runtime Environments in Administrator, expand the runtime environment for the Secure Agent group, click the agent, and copy the agent ID from the browser URL.
All applicable targets

0 COMMENTS

We’d like to hear from you!