Google Cloud Storage V2 Connector

Back Next

Directory-level partitioning for mappings in advanced mode

When you create a mapping in advanced mode, you can read from and write to partition columns.

You can organize tables or data sets into partitions to group the same type of data based on a column or partition key. You can select one or more partition columns in a table or data set.

To read from partition columns, select a partition directory and identify the partition columns. To write to partition columns, you can add partition columns from the list of fields and change the partition order, if required.

You can read data from or write data to partition columns for the following file formats:

Avro

Parquet

ORC

JSON

Importing partition folders

Consider the following rules and guidelines when you import partition folders:

When you read or write data to a partition folder, you can import a directory that contains both partition folder and files. If the directory contains only files but no partition folders, a validation error occurs.

To import a partition directory that contains only partition folders, ensure that the partition directory contains files and data. Otherwise, a validation error is encountered.

You must import a directory that contains only partition folders and select the

Is Directory

option in the advanced source property.

If you import a partition directory that has a partition folder but no files in the partition folder, a validation error is encountered.

When you import a directory that has a partition folder, the data type for the partition column is imported as a String.

When you import a Google Cloud Storage object that has partition columns, the partition fields are listed at the end of the list.

Mappings for Google Cloud Storage

Reading from partition columns

Writing to partition columns

Rules and guidelines for reading from and writing to a partition folder

Download Guide

Watch