Table of Contents

Search

  1. Preface
  2. Introduction to Amazon S3 V2 Connector
  3. Connections for Amazon S3 V2
  4. Amazon S3 V2 sources and targets
  5. Mappings and mapping tasks with Amazon S3 V2
  6. Migrating a mapping
  7. Upgrading to Amazon S3 V2 Connector
  8. Data type reference
  9. Troubleshooting

Amazon S3 V2 Connector

Amazon S3 V2 Connector

Target partitioning

Target partitioning

You can configure partitioning to optimize the mapping performance at run time when you write data to a file of flat format type. You can configure target partitioning only on mappings.
The partition type controls how the agent distributes data among partitions at partition points. You can define the partition type as passthrough partitioning. With partitioning, the Secure Agent distributes rows of target data based on the number of threads that you define as partition.
You can configure the
Merge Partition Files
options in the advanced target properties. You can specify whether the Secure Agent must merge the number of partition files as a single file or maintain separate files based on the number of partitions specified to write data to the Amazon S3 V2 targets.
If you do not select the
Merge Partition Files
option, separate files are created based on the number of partitions specified. The file name is appended with a number starting from 0 in the following format:
<file name>_<number>
For example, the number of threads for the
Region.csv
file is three. If you do not select the
Merge Partition Files
option, the Secure Agent writes three separate files in the Amazon S3 V2 target in the following format:
<Region_0> <Region_1> <Region_2>
If you configure the
Merge Partition Files
option, the Secure Agent merges all the partitioned files as a single file and writes the file to Amazon S3 V2 target.

0 COMMENTS

We’d like to hear from you!