Database Ingestion and Replication

Back Next

Configure a Kafka target

Define target properties for the destination that you selected on the

Destination

page.

Under

Target Properties

, define the following required Kafka target properties:

These properties apply to incremental load operations only.

Property	Description
Use Table Name as Topic Name	Indicates whether Database Ingestion and Replication writes messages that contain source data to separate topics, one for each source table, or writes all messages to a single topic. Select this check box to write messages to separate table-specific topics. The topic names match the source table names, unless you add the source schema name, a prefix, or a suffix in the Include Schema Name , Table Prefix , or Table Suffix properties. By default, this check box is cleared. With the default setting, you must specify the name of the single topic to which all messages are written in the Topic Name property.
Include Schema Name	When Use Table Name as Topic Name is selected, this check box appears and is selected by default. This setting adds the source schema name in the table-specific topic names. The topic names then have the format `schemaname`_`tablename`. If you do not want to include the schema name, clear this check box.
Table Prefix	When Use Table Name as Topic Name is selected, this property appears so that you can optionally enter a prefix to add to the table-specific topic names. For example, if you specify myprefix_, the topic names have the format myprefix_`tablename`. If you omit the underscore (_) after the prefix, the prefix is prepended to the table name.
Table Suffix	When Use Table Name as Topic Name is selected, this property appears so that you can optionally enter a suffix to add to the table-specific topic names. For example, if you specify _mysuffix, the topic names have the format `tablename`_mysuffix. If you omit the underscore (_) before the suffix, the suffix is appended to the table name.
Topic Name	If you do not select Use table name as topic name , you must enter the name of the single Kafka topic to which all messages that contain source data will be written.
Output Format	Select the format of the output file. Options are: CSV AVRO JSON The default value is CSV . Output files in CSV format use double-quotation marks ("") as the delimiter for each field. If your Kafka target uses Confluent Schema Registry to store schemas for incremental load jobs, you must select AVRO as the format.
JSON Format	If JSON is selected as the output format, select the level of detail of the output. Options are: Concise . This format records only the most relevant data in the output, such as the operation type and the column names and values. Verbose . This format records detailed information, such as the table name and column types.
Avro Format	If you selected AVRO as the output format, select the format of the Avro schema that will be created for each source table. Options are: Avro-Flat . This Avro schema format lists all Avro fields in one record. Avro-Generic . This Avro schema format lists all columns from a source table in a single array of Avro fields. Avro-Nested . This Avro schema format organizes each type of information in a separate record. The default value is Avro-Flat .
Avro Serialization Format	If AVRO is selected as the output format, select the serialization format of the Avro output file. Options are: Binary JSON None The default value is Binary . If you have a Confluent Kafka target that uses Confluent Schema Registry to store schemas, select None . Otherwise, Confluent Schema Registry does not register the schema. Do not select None if you are not using Confluent Scheme Registry.
Avro Schema Directory	If AVRO is selected as the output format, specify the local directory where Database Ingestion and Replication stores Avro schema definitions for each source table. Schema definition files have the following naming pattern: `schemaname`_`tablename`.txt If this directory is not specified, no Avro schema definition file is produced. If a source schema change is expected to alter the target, the Avro schema definition file is regenerated with a unique name that includes a timestamp, in the following format: `schemaname`_`tablename`_`YYYYMMDDhhmmss`.txt This unique naming pattern ensures that older schema definition files are preserved for audit purposes.
Avro Compression Type	If AVRO is selected as the output format, select an Avro compression type. Options are: None Bzip2 Deflate Snappy The default value is None , which means no compression is used.
Deflate Compression Level	If Deflate is selected in the Avro Compression Type field, specify a compression level from 0 to 9. The default value is 0.

To view advanced properties, toggle on

Show Advanced Options

. Then under

Advanced Target Properties

, define any of the following optional advanced target properties that you want to use:

Property	Description
Add Operation Type	Select this check box to add a metadata column that includes the source SQL operation type in the output that the job propagates to the target. The job writes "I" for insert, "U" for update, or "D" for delete. By default, this check box is selected.
Add Operation Time	Select this check box to add a metadata column that records the source SQL operation timestamp in the output that the job propagates to the target. By default, this check box is not selected.
Add Operation Owner	Select this check box to add a metadata column that records the owner of the source SQL operation in the output that the job propagates to the target. By default, this check box is not selected. This property is not available for jobs that have a MongoDB or PostgreSQL source. This property is not supported for jobs that have a SQL Server source and use the CDC Tables capture method.
Add Operation Transaction Id	Select this check box to add a metadata column that includes the source transaction ID in the output that the job propagates to the target for SQL operations. By default, this check box is not selected.
Add Orderable Sequence	Select this check box to add a metadata column that records a combined epoch value and an incremental numeric value for each change operation that the job inserts into the target tables. The sequence value is always ascending, but not guaranteed to be sequential and gaps may exist. The sequence value is used to identify the order of activity in the target records. By default, this check box is not selected.
Add Before Images	Select this check box to include UNDO data in the output that a job writes to the target. By default, this check box is not selected.
Async Write	Controls whether to use synchronous delivery of messages to Kafka. Clear this check box to use synchronous delivery. Kafka must acknowledge each message as received before Database Ingestion and Replication sends the next message. In this mode, Kafka is unlikely to receive duplicate messages. However, performance might be slower. Select this check box to use asynchronous delivery. Database Ingestion and Replication sends messages as soon as possible, without regard for the order in which the changes were retrieved from the source. By default, this check box is selected.
Producer Configuration Properties	Specify a comma-separated list of `key`=`value` pairs to enter Kafka producer properties for Apache Kafka, Confluent Kafka, Amazon Managed Streaming for Apache Kafka (MSK), or Kafka-enabled Azure Event Hubs targets. If you have a Confluent target that uses Confluent Schema Registry to store schemas, you must specify the following properties: schema.registry.url=`url`, key.serializer=org.apache.kafka.common.serialization.StringSerializer, value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer You can specify Kafka producer properties in either this field or in the Additional Connection Properties field in the Kafka connection. If you enter the producer properties in this field, the properties pertain to the database ingestion jobs associated with this task only. If you enter the producer properties for the connection, the properties pertain to jobs for all tasks that use the connection definition, unless you override the connection-level properties for specific tasks by also specifying properties in the Producer Configuration Properties field. For information about Kafka producer properties, see the Apache Kafka, Confluent Kafka, Amazon MSK, or Azure Event Hubs documentation.

Under

Table Renaming Rules

, if you want to rename the target objects that are associated with the selected source tables, define renaming rules. Click the + (Add new row) icon and enter a source table name or name mask and enter a corresponding target table name or name mask. To define a mask, include one or more the asterisk (*) wildcards. Then press Enter.

For example, to add the prefix "PROD_" to the names of target tables that correspond to all selected source tables, enter the * wildcard for the source table and enter PROD_* for the target table.

You can enter multiple rules.

Notes:

If you enter the wildcard for a source table mask, you must also enter the wildcard for a target table mask.

If a table name includes special characters, such as a backslash (\), asterisk(*), dot (.), or question mark (?), escape each special character in the name with a backslash (\).

On Windows, if you enter target table renaming criteria that causes a target table name to exceed 232 characters in length, the name is truncated to 222 characters.

Data Ingestion and Replication

appends 14 characters to the name to add a date-time yyyyMMddHHmmss value, which causes the name to exceed the Windows maximum limit of 255. Ensure that the names of any renamed target tables will not exceed 232 characters.

Under

Custom Properties

, you can enter one or more custom properties that Informatica provides to improve performance or to meet your special requirements. To add a property, click the + icon to add a row. In the

Property Name

field, select a property and then enter a property value, or select the

Custom

option and manuallly enter both the property name and value.

The following table describes the properties that are available for this target:

Property	Description
Writer Helper Thread Count	The number of writer helper threads that are used to convert incoming change data rows or initial unload rows to the output format configured for the target, such as Avro, CSV, or Parquet. Default value is 2. If two threads can’t keep up with the incoming volume of data, you can increase the number of threads. Consider increasing the number of threads in the following situations: 1) the incoming volume is high, 2) multiple writer distributors are in use, or 3) some rows are very large or wide, which increases conversion time.
Custom	Select this option to manually enter the name of a property and its value. Use this option to enter properties that Informatica Global Customer Support or a technical staff member has provided to you for a special case. Available for any supported load type.

Property

Description

Writer Helper Thread Count

The number of writer helper threads that are used to convert incoming change data rows or initial unload rows to the output format configured for the target, such as Avro, CSV, or Parquet.

Default value is 2. If two threads can’t keep up with the incoming volume of data, you can increase the number of threads. Consider increasing the number of threads in the following situations: 1) the incoming volume is high, 2) multiple writer distributors are in use, or 3) some rows are very large or wide, which increases conversion time.

Custom

Select this option to manually enter the name of a property and its value. Use this option to enter properties that Informatica Global Customer Support or a technical staff member has provided to you for a special case. Available for any supported load type.

Custom properties are intended to address performance or special processing needs. A property name can contain only alphanumeric characters and the following special characters: periods (.), hyphens (-), and underscores (_).