Communities
A collaborative platform to connect and grow with like-minded Informaticans across the globe
Product Communities
Connect and collaborate with Informatica experts and champions
Discussions
Have a question? Start a Discussion and get immediate answers you are looking for
User Groups
Customer-organized groups that meet online and in-person. Join today to network, share ideas, and get tips on how to get the most out of Informatica
Get Started
Community Guidelines
Knowledge Center
Troubleshooting documents, product guides, how to videos, best practices, and more
Knowledge Base
One-stop self-service portal for solutions, FAQs, Whitepapers, How Tos, Videos, and more
Support TV
Video channel for step-by-step instructions to use our products, best practices, troubleshooting tips, and much more
Documentation
Information library of the latest product documents
Velocity (Best Practices)
Best practices and use cases from the Implementation team
Learn
Rich resources to help you leverage full capabilities of our products
Trainings
Role-based training programs for the best ROI
Certifications
Get certified on Informatica products. Free, Foundation, or Professional
Product Learning Paths
Free and unlimited modules based on your expertise level and journey
Resources
Library of content to help you leverage the best of Informatica products
Tech Tuesdays Webinars
Most popular webinars on product architecture, best practices, and more
Product Availability Matrix
Product Availability Matrix statements of Informatica products
SupportFlash
Monthly support newsletter
Support Documents
Informatica Support Guide and Statements, Quick Start Guides, and Cloud Product Description Schedule
Product Lifecycle
End of Life statements of Informatica products
Ideas
Events
Change Request Tracking
Marketplace
English
  • Data Replication 9.8.0 HotFix 2
  • All Products

Table of Contents

Search

  1. Preface
  2. Data Replication Overview
  3. Understanding Data Replication
  4. Sources - Preparation and Replication Considerations
  5. Targets - Preparation and Replication Considerations
  6. Starting the Server Manager
  7. Getting Started with the Data Replication Console
  8. Defining and Managing Server Manager Main Servers and Subservers
  9. Creating and Managing User Accounts
  10. Creating and Managing Connections
  11. Creating Replication Configurations
  12. Materializing Targets with InitialSync
  13. Scheduling and Running Replication Tasks
  14. Implementing Advanced Replication Topologies
  15. Monitoring Data Replication
  16. Managing Replication Configurations
  17. Handling Replication Environment Changes and Failures
  18. Troubleshooting
  19. Data Replication Files and Subdirectories
  20. Data Replication Runtime Parameters
  21. Command Line Parameters for Data Replication Components
  22. Updating Configurations in the Replication Configuration CLI
  23. DDL Statements for Manually Creating Recovery Tables
  24. Sample Scripts for Enabling or Disabling SQL Server Change Data Capture
  25. Glossary

User Guide

User Guide

Back Next

Defining Apache Kafka Targets

Defining Apache Kafka Targets

To replicate change data to an Apache Kafka target, define target information such as the Kafka topic or topics to which messages will be published, the message format and encoding, and the Avro schema format. You must also specify the Kafka producer properties file that includes connection information.
For Kafka targets, initial synchronization is not necessary.
  1. On the
    Target Database
     tab, click
    New Connection
    . If you want to create a connection based on an existing connection, select a connection in the
    Database Connection
     list before you click
    New connection
    .
    The
    New
     dialog box >
    Database
     view appears.
  2. In the
    Name
     field, enter a name for the new Kafka connection.
    Connection names can contain the digits 0-9, Latin letters A-Z and a-z, and the underscore (_) character. The Data Replication Console truncates connection names that are longer than 120 characters.
  3. In the
    Owner
     list, select an owner for the target definition.
  4. In the
    Type
     list, select
    Apache Kafka
    .
    The following image shows the
    New
     dialog box for a Kafka target: New dialog box view for creating a new target connection.
  5. To the right of the
    Path to Producer Properties File
     field, click
    Browse
     to browse to the Kafka producer.properties file that Data Replication will use to communicate with the Kafka target.
    You must have previously defined the file on the target Server Manager where the Applier runs for Data Replication to access the file. For more information about producer.properties files, see the Apache Kafka documentation.
  6. In the
    Select Topic
     field, select one of the following methods for selecting the Kafka topics to which Data Replication writes messages that contain the source data:
    • Select
      USE_TABLE_NAME
       to direct Data Replication to use the Kafka topic names that match the mapped source table names, for each configuration that uses this connection.
      By default, the auto.create.topics.enable parameter in the Kafka server.properties file is set to true. With this setting, Kafka automatically generates the topic or topics if they do not already exist when messages are sent to the topics. If you set the auto.create.topics.enable parameter to false, Kafka does not generate the topics. You must manually create the topics in Kafka before starting replication based on any configuration that uses this connection.
    • Enter the name of a single Kafka topic to which Data Replication will send the change data from all of the mapped source tables, for each configuration that uses this connection.
    To append additional information from source tables to Kafka topic names, set either of the following advanced runtime parameters on the
    Runtime Settings
     tab >
    Advanced Settings
     view after you finish defining the Kafka target:
    • To include source table schema names in Kafka topic names, set the value of the apply.kafka.kafka_table_name_include_schema parameter to 1.
    • To specify a prefix for the Kafka topic names, enter the prefix in the apply.kafka.kafka_table_name_prefix parameter.
  7. In the
    Message Format
     field, select
    avro
    .
  8. In the
    Message Structure
     field, select the Avro format that Data Replication will use for the messages it sends to target topics. Options are:
    • avroFlatSchemaFormatV1
      . Formats messages in a flat Avro schema format, which lists all Avro fields in one record.
    • avroNestedSchemaFormatV1
      . Formats messages in a nested Avro schema format, which organizes each type of information in a separate record.
  9. In the
    Message Encoding
     field, select the encoding method for Avro messages. Options are:
    • binary
    • json
    If Informatica Intelligent Streaming will consume the data written to the Kafka messaging system, use the binary encoding type. Informatica Intelligent Streaming does not support Kafka messages that use the JSON encoding type.
  10. If you click
    Test Connection
    , Data Replication always reports that the test is successful, even though it cannot validate connectivity to a Kafka target. If the connection information is not valid, Data Replication will report a connection error when it tries to connect to the Kafka target.
  11. Click
    Connect
     to connect to the Kafka target.
    If you connect to the target successfully, the
    Map Tables
     tab opens. Otherwise, Data Replication reports an error.
You can set many advanced runtime parameters for Kafka targets on the
Runtime Settings
 tab >
Advanced Settings
 view. If the default parameter values are not suitable for your environment, edit them. For example, you can customize the checkpoint file name and directory in the apply.kafka.kafka_checkpoint_file_name and apply.kafka.kafka_checkpoint_file_directory parameters.
0 COMMENTS
We’d like to hear from you!