Table of Contents

Search

  1. Preface
  2. Introduction to Amazon S3 V2 Connector
  3. Amazon S3 V2 connections
  4. Amazon S3 V2 sources and targets
  5. Mappings and mapping tasks with Amazon S3 V2
  6. Migrating a mapping
  7. Upgrading to Amazon S3 V2 Connector
  8. Data type reference
  9. Troubleshooting

Amazon S3 V2 Connector

Amazon S3 V2 Connector

Source partitioning

Source partitioning

You can configure fixed partitioning to optimize the mapping performance at run time when you read data from flat, Avro, ORC, or Parquet files. You can configure fixed partitioning only on mappings.
The partition type controls how the agent distributes data among partitions at partition points. With partitioning, the Secure Agent distributes rows of source data based on the number of threads that you define as partition.
Enable partitioning when you configure the Source transformation in the Mapping Designer.
On the
Partitions
tab for the Source transformation, you select fixed partitioning and enter the number of partitions based on the amount of data that you want to read. By default, the value of the
Number of partitions
field is one.
The following image shows the configured partitioning:
On the Partitions tab of the Source transformation, the partitioning type is Fixed and the number of partitions is set to 2.
The Secure Agent enables the partition according to the size of the Amazon S3 V2 source file. The file name is appended with a number starting from 0 in the following format:
<file name>_<number>
If you enable partitioning and the precision for the source column is less than the maximum data length in that column, you might receive unexpected results. To avoid unexpected results, the precision for the source column must be equal to or greater than the maximum data length in that column for partitioning to work as expected.
If you configure partitioning for an Amazon S3 V2 source in a mapping to read from a manifest file, compressed
.gz
file, or a read directory file, the Secure Agent ignores the partition. However, the task runs successfully.

0 COMMENTS

We’d like to hear from you!