Amazon S3 V2 Connector

Back Next

Directory-level partitioning

You can read from and write to partition columns when you use mappings in advanced mode.

You can organize tables or data sets into partitions for grouping same type of data together based on a column or partition key. You can select one or more partition columns in a table or data set.

To read from partition columns, select a partition directory and identify the partition columns. To write to partition columns, you can add partition columns from the list of fields and change the partition order, if required.

You can read data from or write data to partition columns for the following file formats:

Avro

Parquet

ORC

JSON

Reading from partition columns

Perform the following steps to read data from partition columns:

Select a directory from the list of source objects.

Select the Source Type as

Directory

in the Advanced Source Properties.

In the Fields tab, you can view the number of partitions. The

partitionOrder

column appears for the list of partitioned fields, as shown in the following image: The image shows the partition order column.

The

partitionOrder

column specifies whether a column is partitioned.

In the above image, 2 partition columns are present. the partition order values 1 and 2 signify the order in which the

Country

and

State

fields were selected for partitioning. The FileName field has 0 as the partition order.

Writing to partition columns

Perform the following steps to write to partition columns:

Click the

icon in the

Partitions

tab to add the partition columns for a target. The following image shows how you can add the partition columns: The image shows the Add button in the Partitions tab to add the partition columns.