You can configure partitioning to optimize the mapping performance at run time when you write data to Amazon S3 targets.
The partition type controls how the agent distributes data among partitions at partition points. You can define the partition type as passthrough partitioning. With partitioning, the Secure Agent distributes rows of target data based on the number of threads that you define as partition.
You can configure the
Merge Partition Files
options in the advanced target properties. You can specify whether the Secure Agent must merge the number of partition files as a single file or maintain separate files based on the number of partitions specified to write data to the Amazon S3 targets.
If you do not select the
Merge Partition Files
option, separate files are created based on the number of partitions specified. The file name is appended with a number starting from 0 in the following format:
<file name>_<number>
For example, the number of threads for the
Region.csv
file is three. If you do not select the
Merge Partition Files
option, the Secure Agent writes three separate files in the Amazon S3 target in the following format:
<Region_0>
<Region_1>
<Region_2>
If you configure the
Merge Partition Files
option, the Secure Agent merges all the partitioned files as a single file and writes the file to Amazon S3 target.