User Guide

Back Next

Complex File Sources on Azure Blob

You can use a PowerExchange for HDFS or a PowerExchange for Microsoft Azure Blob Storage connection to read data from Azure Blob data objects.

The following table shows the complex files that a mapping can process within Azure Blob storage in the Hadoop environment:

File Type	Supported Formats	Supported Engines
Avro	Flat Hierarchical ¹	Spark
JSON	Flat Hierarchical ¹	Spark
Parquet	Flat Hierarchical ¹	Spark
¹ To run on the Spark engine, the complex file read operation must be enabled to project columns as complex data type.

The following table shows the complex files that a PowerExchange for HDFS connection can process within Azure Blob Storage in the Hadoop environment:

File Type	Supported Formats	Supported Engines
Avro	Flat Hierarchical ^{1 2}	Blaze Spark
JSON	Flat¹ Hierarchical ^{1 2}	Blaze Spark
ORC	Flat	Spark
Parquet	Flat Hierarchical ^{1 2}	Blaze Spark
¹ To run on the Blaze engine, the complex file data object must be connected to a Data Processor transformation. ² To run on the Spark engine, the complex file read operation must be enabled to project columns as complex data type.