Table of Contents

Search

  1. Preface
  2. Introduction to Data Engineering Streaming
  3. Data Engineering Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Appendix A: Connections
  9. Appendix B: Monitoring REST API Reference
  10. Appendix C: Sample Files

Processing Hierarchical Data in Streaming Mappings

Processing Hierarchical Data in Streaming Mappings

Data objects in a streaming mapping can process hierarchical data through complex data types. If you want to project data as complex data types, you must enable the write data operation to project data as complex data types.
The following table shows the format and complex data types that targets in a streaming mapping support:
Object Type
File Type
Supported Formats
Amazon S3
Avro
JSON
Parquet
Hierarchical
Flat
Azure Data Lake Storage Gen1
Avro
Binary
JSON
XML
Flat
Hierarchical
Binary
Azure Data Lake Storage Gen2
Avro
JSON
Parquet
Flat
Hierarchical
Binary
Azure Event Hub
Avro
Binary
JSON
XML
Flat
Hierarchical
Binary
Complex File
Avro
Binary
JSON
XML
Parquet
Flat
Hierarchical
Binary
Confluent Kafka
Avro
Binary
Flat
JSON
XML
Flat
Hierarchical
Binary
Databricks Delta Lake
Avro
Binary
JSON
XML
Flat
Binary
Kafka
Avro
Binary
Flat
JSON
XML
Flat
Hierarchical
Binary
MapR Streams
Avro
Binary
JSON
XML
Flat
Hierarchical
Binary
For more information about processing hierarchical data, see the
Data Engineering Integration User Guide
.