Table of Contents

Search

  1. Preface
  2. Introduction to Data Engineering Streaming
  3. Data Engineering Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Appendix A: Connections
  9. Appendix B: Monitoring REST API Reference
  10. Appendix C: Sample Files

Processing Hierarchical Data in Streaming Mappings

Processing Hierarchical Data in Streaming Mappings

Data objects in a streaming mapping can process hierarchical data through complex data types. If the source contains hierarchical data, you must enable the read data operation to project columns as complex data types.
The following are the complex data types:
  • Array. An array is an ordered collection of elements.
  • Map. A map is an unordered collection of key-value pair elements.
  • Struct. A struct is a collection of elements of different data types.
The following table shows the format and complex data types that sources in a Streaming mapping support:
Format
Schema Type
Amazon Kinesis Streams
Azure Event Hub
JMS
Kafka
MapR Streams
Avro
Flat
Supported
Supported
Not supported
Supported
Supported
Avro
Hierarchical
Supported
Supported
Not supported
Supported
Supported
Binary
Binary
Supported
Supported
Supported
Supported
Supported
Flat
Flat
Supported
Supported
Supported
Supported
Not Supported
JSON
Flat
Supported
Supported
Supported
Supported
Supported
JSON
Hierarchical
Supported
Supported
Supported
Supported
Supported
XML
Flat
Supported
Supported
Supported
Supported
Supported
XML
Hierarchical
Supported
Supported
Supported
Supported
Supported
For more information about processing hierarchical data, see the
Data Engineering Integration User Guide
.