Effective in version 10.2, you can use complex data types, such as array, struct, and map, in mappings that run on the Spark engine. With complex data types, the Spark engine directly reads, processes, and writes hierarchical data in Avro, JSON, and Parquet complex files.
Develop mappings with complex ports, operators, and functions to perform the following tasks:
Generate and modify hierarchical data.
Transform relational data to hierarchical data.
Transform hierarchical data to relational data.
Convert data from one complex file format to another.
When you process hierarchical data, you can use hierarchical conversion wizards to simplify the mapping development tasks. Use these wizards in the following scenarios:
To generate hierarchical data of type struct from one or more ports.
To generate hierarchical data of a nested struct type from ports in two transformations.
To extract elements from hierarchical data in a complex port.
To flatten hierarchical data in a complex port.
For more information, see the "Processing Hierarchical Data on the Spark Engine" chapter in the
