Table of Contents

Search

  1. Preface
  2. Introduction to Big Data Streaming
  3. Big Data Streaming Configuration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Appendix A: Connections
  9. Appendix B: Data Type Reference
  10. Appendix C: Sample Files

Big Data Streaming User Guide

Big Data Streaming User Guide

Kafka Connection

Kafka Connection

The Kafka connection is a Messaging connection. Use the Kafka connection to access an Apache Kafka broker as a source or a target. You can create and manage a Kafka connection in the Developer tool or through infacmd.
The Kafka broker maintains configuration information in Apache ZooKeeper. Apache ZooKeeper is a centralized service that maintains the configuration information of the Apache brokers.
When you configure a Kafka connection, you configure the following properties:
  • The list of Kafka brokers that the connection reads from or writes to.
  • The list of ZooKeeper hosts that maintain the configuration.
  • The number of seconds the Integration Service attempts to reconnect to the database if the connection fails.
  • Version of the Kafka messaging broker. You can select one of the following values:
    • 0.8.x.x-0.10.0.x
    • 0.10.1.x-1.0.0
    You can only use one of the versions at a time in a streaming mapping.