Processing Hierarchical Data on the Spark Engine
Effective in version 10.2.1, the Spark engine includes the following additional functionality to process hierarchical data:
- Map data type
- You can use map data type to generate and process map data in complex files.
- Complex files on Amazon S3
- You can use complex data types to read and write hierarchical data in Avro and Parquet files on Amazon S3. You project columns as complex data type in the data object read and write operations.
For more information, see the "Processing Hierarchical Data on the Spark Engine" chapter in the
Informatica Big Data Management 10.2.1 User Guide.