Rules and Guidelines for Reading from Subdirectories
Rules and Guidelines for Reading from Subdirectories
Consider the following rules and guidelines when you read objects stored in subdirectories:
When you read from or write to a flat file in Microsoft Azure Data Lake Storage Gen2, ensure that the directory or subdirectory name does not contain the percentage (%) character. Else, the mapping fails.
When you run a mapping to read a complex file on the Spark or Databricks Spark engine, ensure that the path specified in the directory override or file name override matches the file path in the source.
You cannot use recursive read when you read from partition columns.
When you read a complex file in a mapping, do not use a tilde (
~
) in the subdirectory name or file name.
When you run a mapping to read a flat file on the Spark or Databricks Spark engine, do not use the following special characters in the directory name or sub-directory name in the directory override:
[] {} " ' + ^ % * ? space
When the FileName port for the source and target is mapped, the file is created in the following format:
When you read a flat file with only headers and no data and map the FileName port, the expected directory structure is not created with the FileName port.