Amazon S3 V2 Connector

Back Next

Reading source objects path

When you import source objects, the Secure Agent appends a FileName field to the imported source object. The FileName field stores the absolute path of the source file from which the Secure Agent reads the data at run time.

For example, a directory contains a number of files and each file contains multiple records that you want to read. You select the directory as source type in the Amazon S3 V2 source advanced properties. When you run the mapping, the Secure Agent reads each record and stores the absolute path of the respective source file in the FileName field.

The FileName field is applicable to the following file formats:

Avro

Binary. Applicable only to mappings.

ORC

Parquet

Avoid using FileName as the column name in the source data. FileName is a reserved keyword. The name is case sensitive.

When you use the FileName field in a source object, the Secure Agent reads file names and directory names differently for mappings and mappings in advanced mode.

Feature	Mapping	Mappings in advanced mode
File name	xyz.amazonaws.com/aa.bb.bucket/1024/characterscheckfor1024	s3a://<bucket_name>/customer.avro
Directory name	<absolute path of the file including the file name>	s3a://<bucket_name>/avro/<directory_name>/<file_name>