Google BigQuery Connectors

Back Next

Key range partitioning

You can configure key range partitioning in a

mapping

that uses the simple or hybrid connection mode to read data from Google BigQuery sources. With key range partitioning, the Secure Agent distributes rows of source data based on the fields that you define as partition keys. The Secure Agent compares the field value to the range values for each partition and sends rows to the appropriate partitions.

Use key range partitioning for columns that have an even distribution of data values. Otherwise, the partitions might have unequal size. For example, a column might have 10 rows between key values 1 and 1000 and the column might have 999 rows between key values 1001 and 2000. If the mapping includes multiple sources, use the same number of key ranges for each source.

When you define key range partitioning for a column, the Secure Agent reads the rows that are within the specified partition range. For example, if you configure two partitions for a column with the ranges as 10 through 20 and 30 through 40, the Secure Agent does not read the rows 20 through 30 because these rows are not within the specified partition range.

You can also use an in-out parameter file to specify the key range values for the partition key columns in a mapping.

You can configure a partition key for fields of the following data types:

String

Integer

Numeric (only if you use a Google BigQuery connection in hybrid mode )

Timestamp

Use the following format: YYYY-MM-DD HH24:MI:SS

Guidelines for key range partitioning in mappings

When you configure key range partitioning in a mapping, consider the following guidelines:

You cannot configure a partition key for Record data type columns and repeated columns.

You cannot configure key range partitioning if a row in the partition key column contains a null value.

You cannot parameterize the value for the partition key column name in the

Partition Key

property.

You cannot parameterize the key range values for a partition key column of the Timestamp data type.

When you run a mapping with

SQL ELT optimization

, you cannot parameterize the key range values.

You cannot use key range partitions when a mapping includes any of the following transformations:

Web Services

JSON to Relational

Partitioning

Download Guide

Watch

Comments

Cloud Connectivity Homepage

Communities

Knowledge Base

Success Portal

0 COMMENTS

We’d like to hear from you! Log in to comment.

Rename Saved Search

Table of Contents

Google BigQuery Connectors

Google BigQuery Connectors

Key range partitioning

Key range partitioning

Guidelines for key range partitioning in mappings