Connections for INFACore

Back Next

Kafka

Create an Kafka connection to read from or write to Kafka.

Feature snapshot

Operation	Support
Read	Yes
Write	Yes

Connection properties

The following table describes the Kafka connection properties:

Property	Description
Connection Name	Name of the connection. The name is not case sensitive. It must be unique within the domain. You can change this property after you create the connection. The name cannot exceed 128 characters, contain spaces, or contain the following special characters: ~ ` ! $ % ^ & * ( ) - + = { [ } ] \| \ : ; " ' < , > . ? /
Kafka Broker List	Comma-separated list of the Kafka brokers. To list a Kafka broker, use the following format: <`HostName`>:<`PortNumber`> When you connect to a Kafka broker over SSL, you must specify the fully qualified domain name for the host name. Otherwise, the test connection fails with SSL handshake error.
Retry Timeout	Optional. Number of seconds after which the Secure Agent attempts to reconnect to the Kafka broker to read or write data. Default is 180 seconds.
Kafka Broker Version	Kafka message broker version. The only valid value is Apache 0.10.1.1 and above.
Additional Connection Properties	Optional. Comma-separated list of additional configuration properties of the Kafka producer or consumer.
Schema Registry URL	Location and port of the Confluent schema registry service to access Avro sources and targets in Kafka. To list a schema registry URL, use the following format: <https>://<`HostName or IP`>:<`PortNumber`> or <http>://<`HostName or IP`>:<`PortNumber`> Example for the schema registry URL: https://kafkarnd.informatica.com:8082 or http://10.65.146.181:8084 Applies only when you import a Kafka topic in Avro format that uses the Confluent schema registry to store the metadata.
SSL Mode	Required. Determines the encryption type to use for the connection. You can choose a mode from the following SSL modes: Disabled. Establishes an unencrypted connection to the Kafka broker. One-way. Establishes an encrypted connection to the Kafka broker using truststore file and truststore password. Two-way. Establishes an encrypted connection to the Kafka broker using truststore file, truststore password, keystore file, and keystore password.
SSL TrustStore File Path	Required when you use the one-way or two-way SSL mode. Absolute path and file name of the SSL truststore file that contains the SSL certificate to connect to the Kafka broker.
SSL TrustStore Password	Required when you use the one-way or two-way SSL mode. Password for the SSL truststore.
SSL KeyStore File Path	Required when you use the two-way SSL mode. Absolute path and file name of the SSL keystore file that contains private keys and certificates to connect to the Kafka broker.
SSL KeyStore Password	Required when you use the two-way SSL mode. Password for the SSL keystore.
Additional Security Properties	Optional. Comma-separated list of additional configuration properties to connect to the Kafka broker in a secure way. If you specify two different values for the same property in Additional Connection Properties and Additional Security Properties , the value in Additional Security Properties overrides the value in Additional Connection Properties .

Schema Registry Security Configuration Properties

When you configure the

Schema Registry URL

connection property, you can configure the schema registry security configuration properties. You can configure one-way SSL, two-way SSL, and basic authentication to connect to the Confluent schema registry in a secure way.

The following table describes the security properties for the Kafka connection when you use the Confluent schema registry:

Property	Description
Additional Security Properties Schema Registry	Optional. Comma-separated list of additional security properties to connect to the Confluent schema registry in a secure way. For example, when you configure basic authentication to establish a secure communication with Confluent schema registry, specify the following value: basic.auth.credentials.source=USER_INFO,basic.auth.user.info=<username>:<password> If you specify two different values for the same property in Additional Connection Properties and Additional Security Properties Schema Registry , the value in Additional Security Properties Schema Registry overrides the value in Additional Connection Properties .

Property

Description

Additional Security Properties Schema Registry

Optional. Comma-separated list of additional security properties to connect to the Confluent schema registry in a secure way.

For example, when you configure basic authentication to establish a secure communication with Confluent schema registry, specify the following value:

basic.auth.credentials.source=USER_INFO,basic.auth.user.info=<username>:<password>

If you specify two different values for the same property in

Additional Connection Properties

and

Additional Security Properties Schema Registry

, the value in

Additional Security Properties Schema Registry

overrides the value in

Additional Connection Properties

Read properties

The following table describes the advanced source properties that you can configure in the Python code to read from Kafka:

Property	Description
Topic Pattern	A regular expression pattern for the topic name that you want to read from. Use the regular expression syntax guidelines to specify the pattern.
Start position offset	The position of the Kafka consumer from where the Kafka Connector starts reading Kafka messages from a Kafka topic. You can select one of the following options: Custom. Read messages from a specific time. Earliest. Read all the messages available on the Kafka topic from the beginning. Latest¹. Read messages received by the Kafka topic after the mapping has been deployed.
Connector Mode	Specifies the mode to read data from the Kafka source. You can select one of the following modes: Batch. The Secure Agent reads the messages available in a Kafka topic based on the position offset you specify. After the Secure Agent reads the data, the mapping terminates. When you configure a mapping for recovery, the Secure Agent reads data from the last checkpoint and stops reading data till the offset when the mapping recovery started. Streaming. The Secure Agent reads the messages available in a Kafka topic in real-time. The Secure Agent continues to read messages from the Kafka topic and does not terminate the mapping. You must manually terminate the mapping task.
Custom Start Position Timestamp	The time in GMT from when the Kafka Connector starts reading Kafka messages from a Kafka topic. Specify the time in the following format: yyyy-mm-dd hh:mm:ss[.fff] The milliseconds are optional. This property is applicable only when you select Custom as the Start position offset.
Consumer Configuration Properties	The configuration properties for the Kafka consumer. Overrides the Additional Connection Properties or Additional Security Properties specified in the Kafka connection. For more information about Kafka consumer configuration properties, see Kafka documentation.

Write properties

The following table describes the advanced target properties that you can configure in the Python code to write to Kafka:

Property	Description
Metadata Fetch Timeout in milliseconds	The time after which the metadata is not fetched. Default value is 10000.
Batch Flush Time in milliseconds	The interval after which the data is published to the target. Default value is 1000.
Batch Flush Size in bytes	The batch size of the events after which the Secure Agent writes data to the target. Default value is 16384.
Producer Configuration Properties	The configuration properties for the producer. Overrides the Additional Connection Properties or Additional Security Properties specified in the Kafka connection. For more information about Kafka producer configuration properties, see Kafka documentation.

Rename Saved Search

Table of Contents

Connections for INFACore

Connections for INFACore

Kafka

Kafka

Feature snapshot

Connection properties

Schema Registry Security Configuration Properties

Read properties

Write properties