Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Big Data Management
  3. Connections
  4. Mappings in the Hadoop Environment
  5. Mapping Objects in the Hadoop Environment
  6. Processing Hierarchical Data on the Spark Engine
  7. Stateful Computing on the Spark Engine
  8. Monitoring Mappings in the Hadoop Environment
  9. Mappings in the Native Environment
  10. Profiles
  11. Native Environment Optimization
  12. Data Type Reference
  13. Complex File Data Object Properties
  14. Function Reference
  15. Parameter Reference

Flat File Sources

Flat File Sources

A mapping that is running in a Hadoop environment can read a flat file source from a native environment.
Consider the following limitations when you configure the mapping to read a flat file source:
  • You cannot use an indirect source type.
  • The row size in a flat file source cannot exceed 190 MB.
  • You cannot use a command to generate or to transform flat file data and send the output to the flat file reader at run time.

Generate the Source File Name

You can generate the source file name for the flat file data object. The content of the file name column remains consistent across different modes of execution.
When you push processing to the specific engine for the required file types, the file name column returns the path based on the following formats:
Pushdown Processing Engine
Type of Files Processes
Returned Path
Hive
HDFS source files
<staged path><HDFS file path>
For example,
hdfs://host name:port/hive/warehouse/ff.txt
Hive
Flat files in the local system
<local file path>
For example,
/home/devbld/Desktop/ff.txt
Blaze
Flat files in the local system
<staged path><local file path>
For example,
hdfs://host name:port/hive/warehouse/home/devbld/Desktop/ff.txt
Spark
HDFS source files
hdfs://<host name>:<port>/<file name path>
For example,
hdfs://host name:port/hive/warehouse/ff.txt
Spark
Flat files in the local system
<local file path>
For example,
/home/devbld/Desktop/ff.txt
The file name column returns the content in the following format for High-Availability cluster:
hdfs://<host name>/<file name path>
For example,
hdfs://irldv:5008/hive/warehouse/ff.txt


Updated December 13, 2018