Table of Contents

Search

  1. Preface
  2. Introduction to Data Engineering Streaming
  3. Data Engineering Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Appendix A: Connections
  9. Appendix B: Monitoring REST API Reference
  10. Appendix C: Sample Files

Kafka Data Objects

Kafka Data Objects

A Kafka data object is a physical data object that represents data in a Kafka stream. After you configure a Messaging connection, create a Kafka data object to write to Apache Kafka brokers.
Kafka runs as a cluster comprised of one or more servers each of which is called a broker. Kafka brokers stream data in the form of messages. These messages are published to a topic. When you write data to a Kafka messaging stream, specify the name of the topic that you publish to. You can also write to a Kerberised Kafka cluster.
Kafka topics are divided into partitions. Spark Structured Streaming can read the partitions of the topics in parallel. This gives better throughput and could be used to scale the number of messages processed. Message ordering is guaranteed only within partitions. For optimal performance you should have multiple partitions.
After you create a Kafka data object, create a write operation. You can use the Kafka data object write operation as a target in Streaming mappings. If you want to configure high availability for the mapping, ensure that the Kafka cluster is highly available.
When you configure the data operation properties, specify the format in which the Kafka data object writes data. You can specify XML, JSON, Avro, or Flat as format. When you specify XML format, you must provide a XSD file. When you specify Avro format, provide a sample Avro schema in a .avsc file. When you specify JSON or Flat format, you must provide a sample file.
You can pass any payload format directly from source to target in Streaming mappings. You can project columns in binary format pass a payload from source to target in its original form or to pass a payload format that is not supported.
Streaming mappings can read, process, and write hierarchical data. You can use array, struct, and map complex data types to process the hierarchical data. You assign complex data types to ports in a mapping to flow hierarchical data. Ports that flow hierarchical data are called complex ports.
For more information about processing hierarchical data, see the
Data Engineering Integration User Guide
.
For more information about Kafka clusters, Kafka brokers, and partitions see http://kafka.apache.org/11/documentation.html.
If you use a Kafka data object in a streaming mapping, you cannot use a MapR Streams data object in the same mapping.

Kafka Combinations Support Matrix

The following table shows the different authentication mechanisms that you can use against listener protocols in a streaming mapping with a Kafka target:
Protocols
Authentication Mechanisms
Plain
GSSAPI
None
SSL+TLS1.0
Yes
Yes
Yes
SSL+TLS1.1
Yes
Yes
Yes
SSL+TLS1.2
Yes
Yes
Yes
SSL+TLS1.0,TLS1.1,TLS1.2
Yes
Yes
Yes
SASL_SSL+TLS1.0
Yes
Yes
NA
SASL_SSL+TLS1.1
Yes
Yes
NA
SASL_SSL+TLS1.2
Yes
Yes
NA
SASL_SSL+TLS1.0,TLS1.1,TLS1.2
Yes
Yes
NA