Table of Contents

Search

  1. Preface
  2. Introduction to Amazon S3 V2 Connector
  3. Connections for Amazon S3 V2
  4. Amazon S3 V2 sources and targets
  5. Mappings and mapping tasks with Amazon S3 V2
  6. Migrating a mapping
  7. Upgrading to Amazon S3 V2 Connector
  8. Data type reference
  9. Troubleshooting

Amazon S3 V2 Connector

Amazon S3 V2 Connector

Writing to multiple target objects

Writing to multiple target objects

When you import target objects, the Secure Agent appends a FileName field to the imported target object. When you map the FileName field in the target object to an incoming field, the Secure Agent creates the folder structure and the target files based on the FileName field. For example:
Syntax:
<tgt_filename_folder>/<tgt_filename=incoming_value_folder>/part_file
Sample:
emp_tgt.parquet/emp_tgt.parquet=128000/part-0000-e9ca8-6af-efd43-455c-8709.c000.parquet
The FileName field is applicable to the following file formats:
  • Avro
  • ORC
  • Parquet
Consider the following guidelines when using the target FileName field in mappings:
  • Do not map the source object FileName field to the target object FileName field. If you map the FileName field in the target object to an incoming field, the Secure Agent does not create directory structure as expected.
  • When you use the FileName field in a target object, the Secure Agent creates folders with the following different names for null values:
    • For mappings:
      _EMPTY_
    • For mappings in advanced mode:
      _HIVE_DEFAULT_PARTITION_
  • When you map a date type incoming field to the FileName field in the target object, the Secure Agent creates a nested folder structure based on the incoming date value for target objects.
  • When you map an incoming field to the FileName field in the target object, the mapping runs successfully for the first time. At subsequent runs, the mapping fails with the following error:
    Operation failed: Index: 0, Size: 0.
    To successfully rerun the
    mapping
    , use a dummy target file at design time and override the dummy target file in advanced target properties.

0 COMMENTS

We’d like to hear from you!