You can configure key range partitioning when you use a mapping to read data from a single Databricks source. The partition type controls how the agent distributes data among partitions at partition points.
You cannot use partitioning in mappings in advanced mode.
Partitioning optimizes the mapping performance at run time. When you run a mapping configured with key range partitioning, the agent distributes rows of source data based on the field that you define as partition keys. The agent compares the field value to the range values for each partition and sends rows to the appropriate partitions.
On the
Partitions
tab for the Source transformation, select key range partitioning and choose the field as the partition key.
You can add key ranges to create partitions, as shown in the following image:
You can configure a partition key for fields of the following data types:
Int, Bigint, Smallint, and Tiny int
Timestamp
Date. Specify the date and time in the following format: YYYY-MM-DD HH24:MI:SS.
Decimal and Double. Specify the range values as whole numbers.
Rules and guidelines for key range partitioning
Consider the following rules and guidelines when you use key range partitioning:
When you use key range partitioning, you can't perform the following operations:
Run a mapping in all-purpose cluster or job cluster.
Use Query or Multiple Objects as the source type.
Configure SQL ELT optimization.
Configure a SQL override.
When a column configured as partition key contains null values, the null values are not written to the target.
ASK INFAPreview
