PowerExchange for Amazon S3 User Guide

10.5
- 10.5.9
- 10.5.8
- 10.5.7
- 10.5.6
- 10.5.5
- 10.5.4
- 10.5.3
- 10.5.2
- 10.5.1
- 10.4.1
- 10.4.0

Back Next

Rules and Guidelines for Using FileName Port

Use the following rules and guidelines when you run a mapping to read or write data using the FileName port:

When you run a mapping to read an Amazon S3 file and if one of the values in the FileName port does not contain any value, the Data Integration Service creates the file in the following format:

However, if you run a mapping to read the newly created file, the mapping fails with the following error message:

java.lang.AssertionError: assertion failed: Empty partition column value in '< >'
at scala.Predef$.assert(Predef.scala:170)

You must ensure that all entries in the FileName port contains a value to read the newly created file successfully.

Do not use a colon (:) and forward slash (/) character in the file name data of the FileName port of the source or target object to run a mapping.

If you connect the FileName port to the target empty zero KB files are created in the target folder.

When you use wildcard character

to read data from a complex file source, the Data Integration Service reads data only from folders or files matching the selection criteria.

For example, if the file path is

/export/home/qamercury/avro/test/cust*

and

Allow Wildcard Characters

option is selected:

This image shows the folders inside a directory

The Data Integration Service ignores all the other folders and only reads

customer.avro

and the files present inside the customer folder.

Do not connect FileName port to a FileName port because the FileName port in the source might contain colon (:) and forward slash (/) characters.

If you create a complex file target in the root directory, map the FileName port to an ID field, and run the mapping in the native environment, the Data Integration Service creates a NULL folder in the root directory and places the target file under the NULL folder.

When you run a mapping in the native environment to read or write an Amazon S3 file and if there are multiple entries with the same name in the source port, you must use a Sorter transformation.

Use the Sorter transformation to sort the source port that you want to map to the FileName port of the Target transformation. After you sort the source port, map the port of the Sorter transformation to the FileName port of the Target transformation. The Data Integration Service creates only one file for each entry with the same name. If you do not use the Sorter transformation, the Data Integration Service creates multiple files for each entry with the same name.

For example, you want to map the following

Employee

source port to the FileName port of the Target transformation and write the data to an Avro target file

target1

Name	ID	SSN
Anna	1	1
John	4	4
Smith	4	4
John	5	5
Anna	2	2

Add a Sorter transformation to sort the source port and map the source port to the port of the Sorter transformation. Then, map the port of the Sorter transformation to the FileName port of the Target transformation. The Data Integration Service creates the following directories along with a single file within each directory:

Directory	Description
target1.avro=Anna	In target1.avro=Anna directory, the Data Integration Service creates one file with the following value: 1,1,1,2,2,2 .
target1.avro=John	In target1.avro=John directory, the Data Integration Service creates one file with the following values: 4,4,4,5,5,5 .
target1.avro=Smith	In target1.avro=Smith directory, the Data Integration Service creates one file with the following values: 4,4,4 .

If you do not add a Sorter transformation, the Data Integration Service creates the following directories along with multiple files within each directories:

Directory	Description
target1.avro=Anna	In target1.avro=Anna directory, the Data Integration Service creates two files with the following value: 1,1,1 and 2,2,2
target1.avro=John	In target1.avro=John directory, the Data Integration Service creates two files with the following value: 4,4,4 and 5,5,5 .
target1.avro=Smith	In target1.avro=Smith directory, the Data Integration Service creates one file with the following value: 4,4,4 .

Rename Saved Search

Table of Contents

PowerExchange for Amazon S3 User Guide

PowerExchange for Amazon S3 User Guide

Rules and Guidelines for Using FileName Port

Rules and Guidelines for Using FileName Port