Hadoop Files V2 Connector

Hadoop Files V2 Connector

Writing to multiple target objects

Writing to multiple target objects

When you import target objects, the Secure Agent appends a FilePath field to the imported target object. When you map the FilePath field in the target object to an incoming field, the Secure Agent creates the folder structure and the target files based on the FilePath field. For example:
Syntax:
<tgt_FilePath_folder>/<tgt_FilePath=incoming_value_folder>/part_file
Sample:
emp_tgt.parquet/emp_tgt.parquet=128000/part-0000-e9ca8-6af-efd43-455c-8709.c000.parquet
The FilePath field is applicable to the following file formats:
  • Avro
  • Parquet
Consider the following guidelines when using the target FilePath field in mappings:
  • Do not map the source object FilePath field to the target object FilePath field. If you map the FilePath field in the target object to an incoming field, the Secure Agent does not create directory structure as expected.
  • When you map a date type incoming field to the FilePath field in the target object, the Secure Agent creates a nested folder structure based on the incoming date value for target objects.

0 COMMENTS

We’d like to hear from you!