Search

Microsoft Azure Data Lake Storage Gen2 Connector

Microsoft Azure Data Lake Storage Gen2 Connector

Back Next

Directory-level partitioning

Directory-level partitioning

You can read from and write to partition columns when you create a mapping in advanced mode.

You can organize tables or data sets into partitions for grouping same type of data together based on a column or partition key. You can select one or more partition columns in a table or data set.

To read from partition columns, select a partition directory and identify the partition columns. To write to partition columns, you can add partition columns from the list of fields and change the partition order, if required.

You can read data from or write data to partition columns for the following file formats:

Avro

Parquet

ORC

JSON

Reading from partition columns

Perform the following steps to read data from partition columns:

Select a directory from the list of source objects. The image shows the list of source objects

The image shows the list of source objects

Select the Source Type as

Directory

in the Advanced Source Properties.

The image shows the directory source types in the advanced source properties

In the Fields tab, you can view the number of partitions.

The

partitionOrder

column indicates whether a column is partitioned and the order in which the fields are selected for partitioning.

The image shows the partition order for the fields

In the above image, 2 partition columns are present. The partition order values 1 and 2 signify the order in which the

Country

and

State

fields were selected for partitioning. The FileName field has 0 as the partition order.

Writing to partition columns

Perform the following steps to write to partition columns:

In the

Partitions

tab, click

Add

to add the partition columns for a target. The image shows the Add option to add the partition columns for a target

The image shows the Add option to add the partition columns for a target

Select the partitioning fields from the list of available fields.

The image shows the list of fields available for partitioning

Click

Select

.

The Partitions tab shows the partition columns that you select.

The image shows the list of partition columns

You can change the partition order using the up and down arrows as shown in the following image:

The image shows the up and down arrows that you can use to change the partition order

Mappings for Microsoft Azure Data Lake Storage Gen2

Rules and guidelines for reading from and writing to a partition folder

Watch

Comments

Cloud Connectivity Homepage

0 COMMENTS

We’d like to hear from you! Log in to comment.