Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Big Data Management
  3. Mappings
  4. Sources
  5. Targets
  6. Transformations
  7. Data Preview
  8. Cluster Workflows
  9. Profiles
  10. Monitoring
  11. Hierarchical Data Processing
  12. Hierarchical Data Processing Configuration
  13. Hierarchical Data Processing with Schema Changes
  14. Intelligent Structure Models
  15. Stateful Computing
  16. Connections
  17. Data Type Reference
  18. Function Reference

Overview of Transformations

Overview of Transformations

Due to the differences between native environments and non-native environments, only certain transformations are valid or are valid with restrictions in a non-native environment. Some functions, expressions, data types, and variable fields are not valid in a non-native environment.
Consider the following processing differences that can affect whether transformations and transformation behavior are valid or are valid with restrictions in a non-native environment:
  • Compute clusters use distributed processing and process data on different nodes. Each node does not have access to the data that is being processed on other nodes. As a result, the run-time engine might not be able to determine the order in which the data originated.
  • Each of the run-time engines in the non-native environment can process mapping logic differently.
  • Much of the processing behavior for batch mappings on the Spark engine also apply to streaming mappings.
The following table lists transformations and support for different engines in a non-native environment:
Transformation
Supported Engines
Transformations not listed in this table are not supported in a non-native environment.
Address Validator
  • Blaze
  • Spark
Aggregator
  • Blaze
  • Spark*
  • Databricks Spark
Case Converter
  • Blaze
  • Spark
Classifier
  • Blaze
  • Spark
Comparison
  • Blaze
  • Spark
Consolidation
  • Blaze
  • Spark
Data Masking
  • Blaze
  • Spark*
Data Processor
  • Blaze
Decision
  • Blaze
  • Spark
Expression
  • Blaze
  • Spark*
  • Databricks Spark
Filter
  • Blaze
  • Spark*
  • Databricks Spark
Java
  • Blaze
  • Spark*
Joiner
  • Blaze
  • Spark*
  • Databricks Spark
Key Generator
  • Blaze
  • Spark
Labeler
  • Blaze
  • Spark
Lookup
  • Blaze
  • Spark*
  • Databricks Spark
Match
  • Blaze
  • Spark
Merge
  • Blaze
  • Spark
Normalizer
  • Blaze
  • Spark*
  • Databricks Spark
Parser
  • Blaze
  • Spark
Python
  • Spark*
Rank
  • Blaze
  • Spark*
  • Databricks Spark
Router
  • Blaze
  • Spark*
  • Databricks Spark
Sequence Generator
  • Blaze
  • Spark
Sorter
  • Blaze
  • Spark*
  • Databricks Spark
Standardizer
  • Blaze
  • Spark
Union
  • Blaze
  • Spark*
  • Databricks Spark
Update Strategy
  • Blaze
  • Spark
Weighted Average
  • Blaze
  • Spark
Window
  • Spark**
* Supported for both batch and streaming mappings.
** Supported for streaming mappings only. For more information, see the Big Data Streaming User Guide.


Updated January 20, 2020