Kafka Connector

Back Next

Kafka targets in mappings

To write data to a Kafka target, configure a Kafka object as the Target transformation in a mapping.

Specify the name and description of Kafka target. Configure the target and advanced properties for the target object in mappings.

The following table describes the target properties that you can configure for a Kafka target:

Property	Description
Connection	Name of the Kafka connection that is associated with a Kafka cluster.
Target Type	Type of the Kafka target objects available. You can write data to a single Kafka target object or parameterize the object. You cannot write data to multiple objects.
Object	Name of the Kafka target object based on the target type selected.
Parameter	Select a parameter for the target object, or click New Parameter to define a new parameter for the target object. The Parameter property appears only if you select Parameter as the target type.
Create New at Runtime²	Creates a target. Enter a name for the target object and path for the target object and select the source fields that you want to use. By default, all source fields are used. The target name can contain alphanumeric characters. You cannot use special characters in the file name except the underscore character (_). When you write data to a field of hierarchical data type, you cannot parameterize the target at runtime.
Format	The file format to write data to a Kafka topic. You can select from the following file format types: None. Writes data to the Kafka topics in binary format. Avro. Writes data to the Kafka topics in Avro format that use the binary encoding type. JSON. Writes data to the Kafka topics in JSON format. To configure the formatting options for the file, click Formatting Options . For more information about format options, see Formatting options for Kafka topics.
Operation	Select Insert as the target operation. You cannot perform update, upsert, or delete operations on a Kafka target topic.
Preview Data	Preview the data of the Kafka target object. When you preview data, Data Integration displays the first 10 rows. By default, Data Integration displays the fields in native order. To display the fields in alphabetical order, enable the Display source fields in alphabetical order option.
¹ Applies only to mappings. ² Applies only to mappings in advanced mode.

The following table describes the advanced properties that you can configure for a Kafka target:

Property	Description
Metadata Fetch Timeout in milliseconds	The time after which the metadata is not fetched. Default value is 10000.
Batch Flush Time in milliseconds	The interval after which the data is published to the target. Default value is 1000.
Batch Flush Size in bytes	The batch size of the events after which the Secure Agent writes data to the target. Default value is 16384.
Producer Configuration Properties	The configuration properties for the producer. Overrides the Additional Connection Properties or Additional Security Properties specified in the Kafka connection. For more information about Kafka producer configuration properties, see Kafka documentation.
Forward Rejected Rows	Not applicable for Kafka Connector.

Rename Saved Search

Table of Contents

Kafka Connector

Kafka Connector

Kafka targets in mappings

Kafka targets in mappings