Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Big Data Management
  3. Connections
  4. Mappings in a Hadoop Environment
  5. Mapping Objects in a Hadoop Environment
  6. Mappings in the Native Environment
  7. Profiles
  8. Native Environment Optimization
  9. Data Type Reference
  10. Function Reference
  11. Parameter Reference

Transformation Support on the Spark Engine

Transformation Support on the Spark Engine

Some restrictions and guidelines apply to processing transformations on the Spark engine.
The following table describes rules and guidelines for the transformations that are supported on the Spark engine:
Transformation
Rules and Guidelines
Aggregator
Mapping validation fails in the following situations:
  • The transformation contains stateful variable ports.
  • The transformation contains unsupported functions in an expression.
Expression
Mapping validation fails in the following situations:
  • The transformation contains stateful variable ports.
  • The transformation contains unsupported functions in an expression.
If an expression results in numerical errors, such as division by zero or SQRT of a negative number, it returns an infinite or an NaN value. In the native environment, the expression returns null values and the rows do not appear in the output.
Filter
Supported without restrictions.
Joiner
Mapping validation fails in the following situations:
  • Case sensitivity is disabled.
Lookup
Mapping validation fails in the following situations:
  • Case sensitivity is disabled.
  • The transformation is not configured to return all rows that match the condition.
  • The lookup is a data object.
  • The lookup is a Hive data source.
  • The cache is configured to be shared, named, persistent, dynamic, or uncached. The cache must be a static cache.
The mapping fails in the following situations:
  • The transformation is unconnected.
Router
Supported without restrictions.
Sorter
Mapping validation fails in the following situations:
  • Case sensitivity is disabled.
The Data Integration Service logs a warning and ignores the Sorter transformation in the following situations:
  • There is a type mismatch in between the target and the Sorter transformation sort keys.
  • The transformation contains sort keys that are not connected to the target.
  • The Write transformation is not configured to maintain row order.
  • The transformation is not directly upstream from the Write transformation.
The Data Integration Service treats null values as high even if you configure the transformation to treat null values as low.
Union
Supported without restrictions.
Transformations not listed in this table are not supported.


Updated July 03, 2018