Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Big Data Management
  3. Connections
  4. Mappings in the Hadoop Environment
  5. Mapping Objects in the Hadoop Environment
  6. Processing Hierarchical Data on the Spark Engine
  7. Stateful Computing on the Spark Engine
  8. Monitoring Mappings in the Hadoop Environment
  9. Mappings in the Native Environment
  10. Profiles
  11. Native Environment Optimization
  12. Data Type Reference
  13. Complex File Data Object Properties
  14. Function Reference
  15. Parameter Reference

Transformations in a Hadoop Environment

Transformations in a Hadoop Environment

Due to the differences between native environment and Hadoop environment, only certain transformations are valid or are valid with restrictions in the Hadoop environment. Some functions, expressions, data types, and variable fields are not valid in the Hadoop environment.
Consider the following processing differences that can affect whether transformations and transformation behavior are valid or are valid with restrictions in the Hadoop environment:
  • Hadoop uses distributed processing and processes data on different nodes. Each node does not have access to the data that is being processed on other nodes. As a result, the Hadoop execution engine might not be able to determine the order in which the data originated.
  • Each of the run-time engines in the Hadoop environment can process mapping logic differently.
The following table lists transformations and levels of support for different engines in a Hadoop environment:
Transformation
Blaze Engine
Spark Engine
Hive Engine
Transformations not listed in this table are not supported in the Hadoop environment.
Address Validator
Restricted
Not supported
Restricted
Aggregator
Restricted
Restricted
Restricted
Case Converter
Supported
Not supported
Supported
Comparison
Supported
Not supported
Supported
Consolidation
Restricted
Not supported
Restricted
Data Masking
Restricted
Restricted
Restricted
Data Processor
Restricted
Not supported
Restricted
Decision
Supported
Not supported
Supported
Expression
Restricted
Restricted
Restricted
Filter
Supported
Supported
Supported
Java
Supported
Supported
Supported
Joiner
Restricted
Restricted
Restricted
Key Generator
Supported
Not supported
Not supported
Labeler
Supported
Not supported
Supported
Lookup
Restricted
Restricted
Restricted
Match
Restricted
Not supported
Restricted
Merge
Supported
Not supported
Supported
Normalizer
Supported
Restricted
Supported
Parser
Supported
Not supported
Supported
Rank
Supported
Restricted
Restricted
Router
Supported
Supported
Supported
Sequence Generator
Restricted
Not supported
Not supported
Sorter
Supported
Restricted
Restricted
Standardizer
Supported
Not supported
Supported
Union
Supported
Supported
Supported
Update Strategy
Restricted
Restricted
Restricted
Weighted Average
Supported
Not supported
Supported


Updated November 09, 2018