Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Big Data Management
  3. Connections
  4. Mappings in the Hadoop Environment
  5. Mapping Objects in the Hadoop Environment
  6. Monitoring Mappings in the Hadoop Environment
  7. Mappings in the Native Environment
  8. Profiles
  9. Native Environment Optimization
  10. Data Type Reference
  11. Function Reference
  12. Parameter Reference
  13. Multiple Blaze Instances on a Cluster

Rules and Guidelines for Mappings in a Hadoop Environment

Rules and Guidelines for Mappings in a Hadoop Environment

You can run mappings in a Hadoop environment. When you run mappings in a Hadoop environment, some differences in processing and configuration apply.
The following processing differences apply to mappings in a Hadoop environment:
  • A mapping is run in high precision mode in a Hadoop environment for Hive 0.11 and above.
  • In a Hadoop environment, sources that have data errors in a column result in a null value for the column. In the native environment, the Data Integration Service does not process the rows that have data errors in a column.
  • When you cancel a mapping that reads from a flat file source, the file copy process that copies flat file data to HDFS may continue to run. The Data Integration Service logs the command to kill this process in the Hive session log, and cleans up any data copied to HDFS. Optionally, you can run the command to kill the file copy process.
  • When you set a limit on the number of rows read from the source for a Blaze mapping, the Data Integration Service runs the mapping with the Hive engine instead of the Blaze engine.
The following configuration differences apply to mappings in a Hadoop environment:
  • Set the optimizer level to none or minimal if a mapping validates but fails to run. If you set the optimizer level to use cost-based or semi-join optimization methods, the Data Integration Service ignores this at run-time and uses the default.
  • Mappings that contain a Hive source or a Hive target must use the same Hive connection to push the mapping to Hadoop.
  • The Data Integration Service ignores the data file block size configured for HDFS files in the hdfs-site.xml file. The Data Integration Service uses a default data file block size of 64 MB for HDFS files. To change the data file block size, copy
    /usr/lib/hadoop/conf/hdfs-site.xml
    to the following location in the Hadoop distribution directory for the Data Integration Service node:
    /opt/Informatica/services/shared/hadoop/[Hadoop_distribution_name]/conf
    . You can also update the data file block size in the following file:
    /opt/Informatica/services/shared/hadoop/[Hadoop_distribution_name]/conf/hive-default.xml
    .


Updated July 03, 2018