Effective in version 10.4.0, the following changes are applicable when you write data to a complex file:
If the mapping source contains null values and you use the
Create Target
option to create a Parquet target file, the default schema contains optional fields and you can insert null values to the target.
Previously, all the fields were created as REQUIRED in the default schema and you had to manually update the data type in the target schema from "required" to "Optional" to write the columns with null values to the target.
If the mapping source contains null values and you use the
Create Target
option to create an Avro target file, null values are defined in the default schema and you can insert null values into the target file.
Previously, the null values were not defined in the default schema and you had to manually update the default target schema to add "null" data type to the schema.
You can manually edit the schema if you do not want to allow null values to the target. You cannot edit the schema to prevent the null values in the target with mapping flow enabled.
These changes are applicable to the following adapters:
PowerExchange for HDFS
PowerExchange for Amazon S3
PowerExchange for Google Cloud Storage
PowerExchange for Microsoft Azure Blob Storage
PowerExchange for Microsoft Azure Data Lake Storage Gen1