PowerExchange for Microsoft Azure Data Lake Storage Gen2 User Guide

10.5
- 10.5.4
- 10.5.3
- 10.5.2
- 10.5.1
- 10.4.1
- 10.4.0

Back Next

Rules and Guidelines for Using FileName Port

Use the following rules and guidelines when you use the FileName data in the FileName port:

To read and write complex files, do not use a colon (:) and forward slash (/) character in the file name data of the FileName port of the source or target object.

To read and write complex files, do not connect FileName port to a FileName port because the FileName port in the source might contain colon (:) and forward slash (/) characters.

When you map the FileName port to an ID field and the ID value contains NULL, the Data Integration Service creates target files with different names in the native and non-native environments.

In the native environment, the target file name is appended with

_EMPTY_

. For example,

target1.avro=_EMPTY_

In the non-native environments, the target file name is appended with

_HIVE_DEFAULT_PARTITION_

. For example,

target1.avro=_HIVE_DEFAULT_PARTITION_

If you create a complex file target in the root directory, map the FileName port to an ID field, and run the mapping in the native environment, the Data Integration Service creates a NULL folder in the root directory and places the target file under the NULL folder.

When you create a mapping to read a flat file, the data preview for the FileName port shows different paths in the native and non-native environments. In the non-native environments, the path also includes ABFSS endpoint details.

In the Native environment, use the Sorter transformation to sort the source port that you want to map to the FileName port of the Target transformation. After you sort the source port, map the port of the Sorter transformation to the FileName port of the Target transformation. The Data Integration Service creates only one file for each value with the same name. If you do not use the Sorter transformation, the Data Integration Service creates multiple files for each value with the same name.

For example, create a mapping in the native environment or on the Spark engine to read or write an Avro file using the FileName port.

The following image shows the Sorter transformation mapping:

This image shows a sample sorter transformation mapping

If you want to map the following source port name to the FileName port of the Target transformation and write the data to an Avro target file

target1

Name	ID	SSN
Anna	1	1
John	4	4
Smith	4	4
John	5	5
Anna	2	2

Add a Sorter transformation to sort the source port and map the source port to the port of the Sorter transformation. Then, map the port of the Sorter transformation to the FileName port of the Target transformation. The Data Integration Service creates the following directories and single file per thread within the directories:

target1.avro=Anna

In this directory, the Data Integration Service creates a file with the following values:

1,1,1,2,2,2

target1.avro=John

In this directory, the Data Integration Service creates a file with the following values:

4,4,4,5,5,5

target1.avro=Smith

In this directory, the Data Integration Service creates a file with the following values:

4,4,4

If you do not add a Sorter transformation, the Data Integration Service creates the following directories and multiple files within the directories:

target1.avro=Anna

In this directory, the Data Integration Service creates two part files with the following values:

1,1,1

and

2,2,2

target1.avro=John

In this directory, the Data Integration Service creates two files with the following values:

4,4,4

and

5,5,5

target1.avro=Smith

In this directory, the Data Integration Service creates one file with the following values:

Rename Saved Search

Table of Contents

PowerExchange for Microsoft Azure Data Lake Storage Gen2 User Guide

PowerExchange for Microsoft Azure Data Lake Storage Gen2 User Guide

Rules and Guidelines for Using FileName Port

Rules and Guidelines for Using FileName Port