Table of Contents

Search

  1. Preface
  2. Introduction to Data Engineering Streaming
  3. Data Engineering Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Appendix A: Connections
  9. Appendix B: Monitoring REST API Reference
  10. Appendix C: Sample Files

Processing Hierarchical Data in Streaming Mappings

Processing Hierarchical Data in Streaming Mappings

Data objects in a streaming mapping can process hierarchical data through complex data types. If the source contains hierarchical data, you must enable the read data operation to project columns as complex data types.
The following are the complex data types:
  • Array. An array is an ordered collection of elements.
  • Map. A map is an unordered collection of key-value pair elements.
  • Struct. A struct is a collection of elements of different data types.
The following table shows the format and complex data types that sources in a streaming mapping support:
Object Type
File Type
Supported Formats
Amazon Kinesis Streams
Avro
Binary
Flat
JSON
XML
Flat
Hierarchical
Binary
Azure Event Hub
Avro
Binary
Flat
JSON
XML
Flat
Hierarchical
Binary
Confluent Kafka
Avro
Binary
Flat
JSON
XML
Flat
Hierarchical
Binary
JMS
Binary
Flat
JSON
Flat
Hierarchical
Binary
Kafka
Avro
Binary
Flat
JSON
XML
Flat
Hierarchical
Binary
MapR Streams
Avro
Binary
JSON
XML
Flat
Hierarchical
Binary
For more information about processing hierarchical data, see the
Data Engineering Integration User Guide
.