PowerExchange for Kafka User Guide for PowerCenter

PowerExchange for Kafka User Guide for PowerCenter

Import Kafka Source and Target Definitions

Import Kafka Source and Target Definitions

Use the
Create PowerExchange for Kafka Source
or
Create PowerExchange for Kafka Target
wizard to import Kafka source and target definitions into the PowerCenter repository.
You must import Kafka source and target objects before you create a mapping.
  1. Start PowerCenter Designer, and connect to a PowerCenter repository configured with a Kafka instance.
  2. Open a source or target folder.
  3. Select
    Source Analyzer
    or
    Target Designer
    .
    • In the Source Analyzer, click
      Sources
      Create PowerExchange for Kafka Source
      as shown in the following image:
      The image shows the location of the Create PowerExchange For Kafka Source option.
      The
      Kafka Connection
      wizard appears.
    • In the Target Analyzer, click
      Targets
      Create PowerExchange for Kafka Target
      as shown in the following image:
      The image shows the location of the Create PowerExchange For Kafka Target option.
      The
      Kafka Connection
      dialog box appears.
  4. Configure the following connection parameters:
    Connection Parameter
    Description
    Kafka Broker List
    The IP address and port combinations of the Kafka messaging system broker list.
    The IP address and port combination has the following format:
    <IP Address>:<port>
    You can enter multiple comma-separated IP address and port combinations.
    Retry Timeout in seconds
    Number of seconds the Integration Service attempts to reconnect to the Kafka broker to write data.
    If the source or target is not available for the time you specify, the mapping execution stops to avoid any data loss.
    Default is 180 seconds.
    Kafka Broker Version
    Select
    Apache 0.10.1.1 & above
    as the Kafka messaging broker version.
    Additional Connection Properties
    Comma-separated list of connection properties to connect to the Kafka broker.
    To connect to a Kafka broker over SSL, ensure that you set the following properties:
    • security.protocol
    • ssl.truststore.location
    • ssl.truststore.password
    If the
    ssl.client.auth
    parameter is set to
    required
    or
    requested
    , ensure that you set the following properties:
    • ssl.keystore.location
    • ssl.keystore.password
    • ssl.key.password
    For example,
    security.protocol=SSL,ssl.truststore.location=<complete path and name of the client.truststore.jks file>,ssl.truststore.password=<client truststore password>,ssl.keystore.location=<complete path and name of the client.keystore.jks file>,ssl.keystore.password=<client keystore password>,ssl.key.password=<password for the individual key>
    You can also specify the configuration properties required to connect to a Kerberised Kafka cluster.
    Specify the following property:
    security.protocol=SASL_PLAINTEXT,sasl.kerberos.service.name=kafka,sasl.mechanism=GSSAPI
    Confluent Schema Registry URL
    Address that points to your Schema Registry instance to manage schemas within the Confluent platform. For example,
    https://<host name>:8081
    .
    SSL Mode
    Specifies whether the PowerCenter Integration Service establishes a secure connection to the Kafka broker. You can select one of the following options:
    • disabled. The PowerCenter Integration Service establishes an unencrypted connection to the Kafka broker.
    • One-Way. The PowerCenter Integration Service establishes an encrypted connection to the Kafka broker using truststore file and truststore password.
    • Two-Way. The PowerCenter Integration Service establishes an encrypted connection to the Kafka broker using truststore file and truststore password.
    SSL TrustStore File Path
    Applicable only if you select
    One-Way
    or
    Two-Way
    as the SSL mode.
    The complete path and file name of the truststore file. The truststore file contains the SSL certificate that the Kafka cluster validates against the Kafka broker certificate.
    SSL TrustStore Password
    Applicable only if you select
    One-Way
    or
    Two-Way
    as the SSL mode.
    The password for the truststore file.
    SSL KeyStore File Path
    Applicable only if you select
    Two-Way
    as the SSL mode.
    The complete path and name of the Java keystore file. The keystore file contains the certificate that the Kafka broker validates against the Kafka cluster certificate.
    SSL KeyStore Password
    Applicable only if you select
    Two-Way
    as the SSL mode.
    The password for the keystore file.
    Additional Security Properties
    Not applicable for PowerExchange for Kafka.
    SSL Mode Schema Registry
    Applicable for schema registry. Specifies whether the PowerCenter Integration Service establishes a secure connection to the Kafka broker. You can select one of the following options:
    • disabled. The PowerCenter Integration Service establishes an unencrypted connection to the Kafka broker.
    • One-Way. The PowerCenter Integration Service establishes an encrypted connection to the Kafka broker using truststore file and truststore password.
    • Two-Way. The PowerCenter Integration Service establishes an encrypted connection to the Kafka broker using truststore file and truststore password.
    SSL TrustStore File Path Schema Registry
    Applicable only if you select
    One-Way
    or
    Two-Way
    as the SSL mode for schema registry.
    The complete path and file name of the truststore file. The truststore file contains the SSL certificate that the Kafka cluster validates against the Kafka broker certificate.
    SSL TrustStore Password Schema Registry
    Applicable only if you select
    One-Way
    or
    Two-Way
    as the SSL mode for schema registry.
    The password for the truststore file.
    SSL KeyStore File Path Schema Registry
    Applicable only if you select
    Two-Way
    as the SSL mode for schema registry.
    The complete path and name of the Java keystore file. The keystore file contains the certificate that the Kafka broker validates against the Kafka cluster certificate.
    SSL KeyStore Password Schema Registry
    Applicable only if you select
    Two-Way
    as the SSL mode for schema registry.
    The password for the keystore file.
    Additional Security Properties Schema Registry
    Applicable for schema registry. For Avro schema registry, enter the following security property:
    basic.auth.credentials.source=USER_INFO, basic.auth.user.info=user:password
  5. Click
    Test
    to test the connection.
  6. Click
    Finish
    to add the connection.
    The
    Select Objects from
    tab appears.
  7. Select the
    Topics
    package in
    Package Explorer
    .
    A list of topic appears as shown in the following image:
    The image shows the Package Explorer tab where a list of Kafka topics appear.
  8. Select the topic that you want to import, and then click
    Next
    .
    You cannot select topics from the
    Topics Pattern
    package.
    The
    Select Format Type for
    dialog box appears.
    The following image shows the
    Select Format Type for
    dialog box:
    The image shows the Package Explorer tab where a list of table appears after you select the dataset.
  9. In the
    Format Type
    list, select
    Json
    ,
    Avro
    , or
    Binary
    to import data from Kafka topics in JSON format or Avro format, or Binary format.
    You cannot import a PowerExchange for Kafka source or target definition in Flat or XML format.
    By default, the system sets
    Binary
    as the format type. The
    Avro
    file type supports binary encoding.
  10. If you select
    Json
    ,
    Avro
    , or
    Binary
    as the
    Format Type
    , you must configure the following format properties:
    Property
    Description
    Schema Source
    Specifies the mode to import schema for the Kafka topic.
    Select
    Import Schema from File
    to import schema from a schema definition file in your local machine.
    You cannot import a schema for a Kafka topic using the
    Read Data from File
    option.
    Schema
    When you select the
    Import Schema from File
    option in the
    Schema Source
    drop-down list, you must choose a schema definition file on your local machine.
  11. Optionally, click
    Data Preview
    to preview data.
    To view the topic metadata, select the topic, and double-click the topic name.
    You cannot select multiple topics in a Kafka source or target definition.
  12. Click
    Import
    .
    If you click
    Import
    in the
    Select Objects from
    dialog box, PowerExchange for Kafka imports the data from the Kafka topic in binary format.

0 COMMENTS

We’d like to hear from you!