Mappings

Back Next

Overriding source partitions

If you use partitions with a parameterized source connection or source object, you can override the partitions using a parameter file. To override the partitions, create a JSON file that defines the new partition configuration and specify that file in the parameter file.

You can override partitions for flat file and relational connections.

In the mapping, ensure that either the source connection parameter or the source object parameter can be overridden at runtime.

Configure default partitions in the mapping or the

mapping

task.

Create a JSON file that defines the new partition configuration. For each Source transformation, add an object with the following structure:

{
    "name": "$<Source transformation name>$",
    "partitions": {
        <partition details>
    }
}

If you're configuring partitions for a Source transformation in a mapplet, include

Mapplet_

before the transformation name. For example:

"name": "$Mapplet_MySource$"

Add partition details based on the partitioning method.

You can can override the partitioning method if needed. The method you set in the JSON file doesn't need to match the method in the mapping or

mapping

task.

For fixed partitioning, define the number of partitions in the following format:

"partitions": {
    "type": "PASS_THROUGH",
    "numOfPartitions": <number of partitions>
}

For key range partitioning, provide the source table name, partition key field name, and key ranges in the following format:

"partitions": {
    "type": "KEY_RANGE",
    "tableName": "<source table name>",
    "columnName": "<partition key field name>",
    "ranges": [
        {
            "start": <value A>,
            "end": <value B>
        }
    ]
}

Include as many objects with start and end values as needed in the

ranges

array. To use an undefined minimum value for the first key range, set the start value to an empty string. To use an undefined maximum value for the last key range, set the end value to an empty string.

For pass-through partitioning, define an SQL query or source filter for each partition in the following format:

"partitions": {
    "type": "PASS_THROUGH_STRATEGY",
    "passThroughAttributes": [
        [
            {
                "name": <either "SQL Query" or "Source Filter">,
                "value": <query or filter string>
            }
        ]
    ]
}

Include as many query or filter objects as needed in the

passThroughAttributes

array.

For more information about partitioning, see "Partitions" in the "Source transformation" chapter in

Transformations

Save the JSON file in the same directory as the parameter file.

In the parameter file, specify the JSON file with the following parameter and value:

$$mtTaskOverride.path=FILE:<file name>

If the parameter file uses sections, add this parameter in a task section. If you define this parameter in the global section, a taskflow section, or a linear taskflow section,

Data Integration

ignores it.

If you want to change the partition configuration, update the JSON file.

JSON file example

The following example shows a JSON file that overrides fixed partitioning in the Source transformation Source1 and pass-through partitioning in the Source transformation Source2:


[
    {
        "name": "$Source1$",
        "partitions": {
            "type": "PASS_THROUGH",
            "numOfPartitions": 3
        }
    },
    {
        "name": "$Source2$",
        "partitions": {
            "type": "PASS_THROUGH_STRATEGY",
            "passThroughAttributes": [
                [
                    {
                        "name": "SQL Query",
                        "value": "SELECT * FROM DEPT_DATA WHERE DEPTNO>20"
                    }
                ],
                [
                    {
                        "name": "Source Filter",
                        "value": "LOC='DALLAS'"
                    }
                ],
                [
                    {
                        "name": "SQL Query",
                        "value": "SELECT * FROM DEPT_DATA WHERE DNAME='RESEARCH'"
                    }
                ]
            ]
        }
    }
]

Using parameter files with mapping tasks

Download Guide

Watch

Comments

Cloud Data Integration Homepage