Table of Contents

Search

  1. Preface
  2. Introduction to PowerExchange for Microsoft Azure Data Lake Storage Gen2
  3. PowerExchange for Microsoft Azure Data Lake Storage Gen2 Configuration
  4. Microsoft Azure Data Lake Storage Gen2 Connections
  5. PowerExchange for Microsoft Azure Data Lake Storage Gen2 Data Objects
  6. Microsoft Azure Data Lake Storage Gen2 Mappings
  7. Appendix A: Microsoft Azure Data Lake Storage Gen2 Datatype Reference

PowerExchange for Microsoft Azure Data Lake Storage Gen2 User Guide

PowerExchange for Microsoft Azure Data Lake Storage Gen2 User Guide

Rules and Guidelines for Reading from Subdirectories

Rules and Guidelines for Reading from Subdirectories

Consider the following rules and guidelines when you read objects stored in subdirectories:
  • When you read from or write to a flat file in Microsoft Azure Data Lake Storage Gen2, ensure that the directory or subdirectory name does not contain the percentage (%) character. Else, the mapping fails.
  • When you run a mapping to read a complex file on the Spark or Databricks Spark engine, ensure that the path specified in the directory override or file name override matches the file path in the source.
  • You cannot use recursive read when you read from partition columns.
  • When you read a complex file in a mapping, do not use a tilde (
    ~
    ) in the subdirectory name or file name.
  • When you run a mapping to read a flat file on the Spark or Databricks Spark engine, do not use the following special characters in the directory name or sub-directory name in the directory override:
    [] {} " ' + ^ % * ? space
  • When the FileName port for the source and target is mapped, the file is created in the following format:
    target_filename=source_directory_subdirectory1_sudirectory2
  • When you read a flat file with only headers and no data and map the FileName port, the expected directory structure is not created with the FileName port.

0 COMMENTS

We’d like to hear from you!