Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Data Engineering Integration
  3. Mappings
  4. Mapping Optimization
  5. Sources
  6. Targets
  7. Transformations
  8. Python Transformation
  9. Data Preview
  10. Cluster Workflows
  11. Profiles
  12. Monitoring
  13. Hierarchical Data Processing
  14. Hierarchical Data Processing Configuration
  15. Hierarchical Data Processing with Schema Changes
  16. Intelligent Structure Models
  17. Blockchain
  18. Stateful Computing
  19. Appendix A: Connections Reference
  20. Appendix B: Data Type Reference
  21. Appendix C: Function Reference

Overview of Transformations

Overview of Transformations

Due to the differences between native environments and non-native environments, only certain transformations are valid or are valid with restrictions in a non-native environment. Some functions, expressions, data types, and variable fields are not valid in a non-native environment.
Consider the following processing differences that can affect whether transformations and transformation behavior are valid or are valid with restrictions in a non-native environment:
  • Compute clusters use distributed processing and process data on different nodes. Each node does not have access to the data that is being processed on other nodes. As a result, the run-time engine might not be able to determine the order in which the data originated.
  • Each of the run-time engines in the non-native environment can process mapping logic differently.
  • Much of the processing behavior for batch mappings on the Spark engine also apply to streaming mappings.
  • You can access Delta Lake tables with any transformation that runs on Databricks Spark.
The following table lists transformations and support for different engines in a non-native environment:
Transformation
Supported Engines
Transformations not listed in this table are not supported in a non-native environment.
Address Validator
  • Blaze
  • Spark
Aggregator
  • Blaze
  • Spark*
  • Databricks Spark
Case Converter
  • Blaze
  • Spark
Classifier
  • Blaze
  • Spark
Comparison
  • Blaze
  • Spark
Consolidation
  • Blaze
  • Spark
Data Masking
  • Blaze
  • Spark*
Data Processor
  • Blaze
  • Spark**
Decision
  • Blaze
  • Spark
Expression
  • Blaze
  • Spark*
  • Databricks Spark
Filter
  • Blaze
  • Spark*
  • Databricks Spark
Java
  • Blaze
  • Spark*
Joiner
  • Blaze
  • Spark*
  • Databricks Spark
Key Generator
  • Blaze
  • Spark
Labeler
  • Blaze
  • Spark
Lookup
  • Blaze
  • Spark*
  • Databricks Spark
Match
  • Blaze
  • Spark
Merge
  • Blaze
  • Spark
Normalizer
  • Blaze
  • Spark*
  • Databricks Spark
Parser
  • Blaze
  • Spark
Python
  • Spark*
  • Databricks Spark
Rank
  • Blaze
  • Spark*
  • Databricks Spark
Router
  • Blaze
  • Spark*
  • Databricks Spark
Sequence Generator
  • Blaze
  • Spark
Sorter
  • Blaze
  • Spark*
  • Databricks Spark
Standardizer
  • Blaze
  • Spark
Union
  • Blaze
  • Spark*
  • Databricks Spark
Update Strategy
  • Blaze
  • Spark
  • Databricks Spark
Weighted Average
  • Blaze
  • Spark
Window
  • Spark***
* Supported for both batch and streaming mappings.
** Supported with restrictions in batch mappings. Not supported in streaming mappings. For information about the Data Processor transformation support on the Spark engine, see the KB article.
*** Supported for streaming mappings only. For more information, see the
Data Engineering Streaming User Guide
.


Updated September 28, 2020