Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Big Data Management
  3. Connections
  4. Mappings in the Hadoop Environment
  5. Mapping Objects in the Hadoop Environment
  6. Monitoring Mappings in the Hadoop Environment
  7. Mappings in the Native Environment
  8. Profiles
  9. Native Environment Optimization
  10. Data Type Reference
  11. Function Reference
  12. Parameter Reference

Flat File Sources

Flat File Sources

A mapping that is running in a Hadoop environment can read a flat file source from a native environment.
Consider the following limitations when you configure the mapping to read a flat file source:
  • You cannot use an indirect source type.
  • The row size in a flat file source cannot exceed 190 MB.
  • You cannot use a command to generate or to transform flat file data and send the output to the flat file reader at run time.

Generate the Source File Name

You can generate the source file name for the flat file data object. The content of the file name column remains consistent across different modes of execution.
When you push processing to the specific engine for the required file types, the file name column returns the path based on the following formats:
Pushdown Processing Engine
Type of Files Processes
Returned Path
Hive
HDFS source files
<staged path><HDFS file path>
For example,
hdfs://host name:port/hive/warehouse/ff.txt
Hive
Flat files in the local system
<local file path>
For example,
/home/devbld/Desktop/ff.txt
Blaze
Flat files in the local system
<staged path><local file path>
For example,
hdfs://host name:port/hive/warehouse/home/devbld/Desktop/ff.txt
Spark
HDFS source files
hdfs://<host name>:<port>/<file name path>
For example,
hdfs://host name:port/hive/warehouse/ff.txt
Spark
Flat files in the local system
<local file path>
For example,
/home/devbld/Desktop/ff.txt
The file name column returns the content in the following format for High-Availability cluster:
hdfs://<host name>/<file name path>
For example,
hdfs://irldv:5008/hive/warehouse/ff.txt


Updated November 09, 2018