Table of Contents

Search

  1. Preface
  2. Introduction to Data Engineering Streaming
  3. Data Engineering Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Appendix A: Connections
  9. Appendix B: Monitoring REST API Reference
  10. Appendix C: Sample Files

Azure Event Hubs Data Objects

Azure Event Hubs Data Objects

An Azure Event Hubs data object is a physical data object that represents data in Microsoft Azure Event Hubs data streaming platform and event ingestion service. After you create an Azure Event Hubs connection, create an Azure Event Hubs data object to write to an event hub event.
Azure Event Hubs is a highly scalable data streaming platform and event ingestion service, that receives and processes events. Azure Event Hubs can process and store events or data produced by distributed software and devices.
When you configure the Azure Event Hubs data object, specify the name of the event that you write to. After you create the data object, create a data object write operation to write data to an event hub. You can then add the data object write operation as a target in streaming mappings.
Configure partition keys to separate events into different partitions. Events with the same partition key are sent to the same partition on the event hub. Events that are sent in a batch do not need to have the same partition key.
When you configure the data operation properties, specify the format in which the Azure Event Hubs data object writes data. You can specify XML, JSON, Avro, or Flat as format. When you specify XML format, you must provide a XSD file. When you specify Avro format, provide a sample Avro schema in a .avsc file. When you specify JSON or Flat format, you must provide a sample file.
You can pass any payload format directly from source to target in Streaming mappings. You can project columns in binary format pass a payload from source to target in its original form or to pass a payload format that is not supported.
Streaming mappings can read, process, and write hierarchical data. You can use array, struct, and map complex data types to process the hierarchical data. You assign complex data types to ports in a mapping to flow hierarchical data. Ports that flow hierarchical data are called complex ports.
For more information about processing hierarchical data, see the
Data Engineering Integration User Guide
.