Table of Contents

Search

  1. Preface
  2. Introduction to Microsoft Azure Data Lake Storage Gen2 Connector
  3. Connections for Microsoft Azure Data Lake Storage Gen2
  4. Mappings for Microsoft Azure Data Lake Storage Gen2
  5. Migrating a mapping
  6. Data type reference
  7. Troubleshooting

Microsoft Azure Data Lake Storage Gen2 Connector

Microsoft Azure Data Lake Storage Gen2 Connector

Rules and guidelines for FileName field

Rules and guidelines for FileName field

Consider the following guidelines when you use the FileName field in mappings:
  • FileName is a reserved keyword. Avoid using FileName as the column name in the source data. The name is case sensitive.
  • Do not map the source object FileName field to the target object FileName field for a complex file. If you map the FileName field in the target object to an incoming field, the Secure Agent does not create directory structure as expected.
  • When you use the FileName field in a target object, the Secure Agent creates folders with different names for null values for mappings in advanced mode:
    • For mappings, the target file name is appended with
      _EMPTY_
      .
    • For mappings in advanced mode, the target file name is appended with
      _HIVE_DEFAULT_PARTITION_
  • When you map a date type incoming field to the FileName field in the target object, the Secure Agent creates a nested folder structure based on the incoming date value for target objects.
  • When you map an incoming field to the FileName field in the target object and the target has the
    Handle Special Characters
    option enabled or the target file name has special characters, the mapping fails with the following error on a Windows machine:
    [ERROR] java.io.IOException: The filename, directory name, or volume label syntax is incorrect
  • When you map an incoming field to the FileName field in the target object, the mapping runs successfully for the first time. At subsequent runs, the mapping fails with the following error:
    Object not found.
    To successfully rerun the mapping, use a dummy target file at design time and override the dummy target file in advanced target properties.
  • When you create a target at runtime, the target file name is not generated in the expected format.
    This issue occurs if the FileName field is enabled for the target object. To resolve this issue, exclude the FileName field from the incoming fields in target.
  • When you write data to an existing target object or create a new target object with a partition directory, the FileName field is not added on the target side. The FileName field is only present at the source side.
    To read the FileName field data from the source, use the Expression transformation to rename the FileName field to a different name to avoid any validation failures.

0 COMMENTS

We’d like to hear from you!