Amazon S3 V2 Connector

Back Next

File formatting options

When you select the format of an Amazon S3 file, you can configure the formatting options.

The following table describes the formatting options for Avro, Parquet, JSON, ORC, and delimited flat files:

Property	Description
Schema Source	The schema of the source or target file. You can select one of the following options to specify a schema: Read from data file. Imports the schema from the file in Amazon S3. Import from Schema File. Imports schema from a schema definition file in your local machine.
Schema File	Upload a schema definition file. You cannot upload a schema file when you create a target at runtime.

The following table describes the formatting options for flat files:

Property	Description
Read from data file	Imports the schema from the file in Amazon S3. If you select Read from data file and use the JSON² file format, you can select one of the following options: Data elements to sample. The number of rows to read from the metadata. Memory available to process data. The memory that the parser uses to read the JSON sample schema and process it. You can increase the parser memory. Default is 2 MB.
Import from schema file	Imports schema from a schema definition file in your local machine. If you select Import from schema file , you can select Schema File to upload a schema file. You cannot upload a schema file when you select the Create Target option to write data to Amazon S3.
Flat File Type	The type of flat file. Select one of the following options: Delimited. Reads a flat file that contains column delimiters. Fixed Width. Reads a flat file with fields that have a fixed length. You must select the file format in the Fixed Width File Format option. If you do not have a fixed-width file format, click New Components Fixed Width File Format to create one.
Delimiter	Character used to separate columns of data. You can configure parameters such as comma, tab, colon, semicolon, or others. To set a tab as a delimiter, you must type the tab character in any text editor. Then, copy and paste the tab character in the Delimiter field.
Escape Char	Character immediately preceding a column delimiter character embedded in an unquoted string, or immediately preceding the quote character in a quoted string. You can specify a character or \<decimal value> . When you specify \<decimal value> , the agent considers the ASCII character for the decimal value as the escape character. For example, if you specify \64 , the agent considers the ASCII character @ . To ignore the escape character, specify \0 .
Qualifier	Quote character that defines the boundaries of data. You can set the qualifier as single quote or double quote.
Qualifier Mode	Specify the qualifier behavior for the target object. You can select one of the following options: Minimal. Applies qualifier to data that has a delimiter value in the data. Otherwise, the Secure Agent does not apply the qualifier when writing data to the target. All. Applies the qualifier to all non-empty columns. Default mode is minimal.
Code Page	Select the code page that the agent must use to read or write data. Amazon S3 V2 Connector supports the following code pages: MS Windows Latin 1. Select for ISO 8859-1 Western European data. UTF-8. Select for Unicode and non-Unicode data. Shift-JIS. Select for double-byte character data. ISO 8859-15 Latin 9 (Western European). ISO 8859-2 Eastern European. ISO 8859-3 Southeast European. ISO 8859-5 Cyrillic. ISO 8859-9 Latin 5 (Turkish). IBM EBCDIC International Latin-1.
Disable escape char when a qualifier is set	Check to disable the escape character when a qualifier value is already set.
Header Line Number	Specify the line number that you want to use as the header when you read data from Amazon S3. You can also read a file that does not have a header. To read data from a file with no header, specify the value of the Header Line Number field as 0. To read data from a file with a header, set the value of the Header Line Number field to a value that is greater than or equal to one. This property is applicable when you preview the source data and at runtime for the mapping. Default is 1.
First Data Row¹	Specify the line number from where you want the Secure Agent to read data. You must enter a value that is greater or equal to one. To read data from the header, the value of the Header Line Number and the First Data Row fields should be the same. Default is 1. This property is applicable during runtime and data preview to read a file. This property is applicable during data preview to write a file.
Target Header	Select whether you want to write data to a target that contains a header or without a header in the flat file. You can select With Header or Without Header options. This property is not applicable when you read data from a Amazon S3 source.
Distribution Column¹	Specify the name of the column that is used to create multiple target files during run time. This property is not applicable when you read data from a Amazon S3 source. For more information about the distribution column, see Distribution column.
Max Rows To Preview	Not applicable to Amazon S3 V2 Connector.
Row Delimiter	Character used to separate rows of data. You can set values as \r, \n, and \r\n.
¹Doesn't apply to mapping in advanced mode. ²Applies only to mappings in advanced mode.

The following table describes the formatting options for JSON files:

Property	Description
Data elements to sample¹	Specify the number of rows to read to find the best match to populate the metadata.
Memory available to process data¹	The memory that the parser uses to read the JSON sample schema and process it. The default value is 2 MB. If the file size is more than 2 MB, you might encounter an error. Set the value to the file size that you want to read.
Read multiple-line JSON files	Not applicable.
¹Applies only to mappings in advanced mode.