Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Big Data Management
  3. Connections
  4. Mappings in the Hadoop Environment
  5. Mapping Objects in the Hadoop Environment
  6. Monitoring Mappings in the Hadoop Environment
  7. Mappings in the Native Environment
  8. Profiles
  9. Native Environment Optimization
  10. Data Type Reference
  11. Function Reference
  12. Parameter Reference
  13. Multiple Blaze Instances on a Cluster

Transformations in a Hadoop Environment

Transformations in a Hadoop Environment

Due to the differences between native environment and Hadoop environment, only certain transformations are valid or are valid with restrictions in the Hadoop environment. Some functions, expressions, data types, and variable fields are not valid in the Hadoop environment.
Consider the following processing differences that can affect whether transformations and transformation behavior are valid or are valid with restrictions in the Hadoop environment:
  • Hadoop uses distributed processing and processes data on different nodes. Each node does not have access to the data that is being processed on other nodes. As a result, the Hadoop execution engine might not be able to determine the order in which the data originated.
  • Each of the run-time engines in the Hadoop environment can process mapping logic differently.
The following table lists transformations and levels of support for different engines in a Hadoop environment:
Transformation
Blaze Engine
Spark Engine
Hive Engine
Transformations not listed in this table are not supported in the Hadoop environment.
Address Validator
Supported with restrictions
Not supported
Supported with restrictions
Aggregator
Supported with restrictions
Supported with restrictions
Supported with restrictions
Case Converter
Supported
Not supported
Supported
Comparison
Supported
Not supported
Supported
Consolidation
Supported with restrictions
Not supported
Supported with restrictions
Data Masking
Supported with restrictions
Not supported
Supported with restrictions
Data Processor
Supported with restrictions
Not supported
Supported with restrictions
Decision
Supported
Not supported
Supported
Expression
Supported with restrictions
Supported with restrictions
Supported with restrictions
Filter
Supported
Supported
Supported
Java
Supported
Supported with restrictions
Supported with restrictions
Joiner
Supported with restrictions
Supported with restrictions
Supported with restrictions
Key Generator
Supported
Not supported
Not supported
Labeler
Supported
Not supported
Supported
Lookup
Supported with restrictions
Supported with restrictions
Supported with restrictions
Match
Supported with restrictions
Not supported
Supported with restrictions
Merge
Supported
Not supported
Supported
Normalizer
Supported
Not supported
Supported
Parser
Supported
Not supported
Supported
Rank
Supported
Not supported
Supported with restrictions
Router
Supported
Supported
Supported
Sequence Generator
Supported with restrictions
Not supported
Not supported
Sorter
Supported
Supported with restrictions
Supported with restrictions
SQL
Not supported
Not supported
Supported with restrictions
Standardizer
Supported
Not supported
Supported
Union
Supported
Supported
Supported
Update Strategy
Supported with restrictions
Not supported
Supported with restrictions
Weighted Average
Supported
Not supported
Supported


Updated July 03, 2018