Table of Contents

Search

  1. Preface
  2. Introduction to Intelligent Streaming
  3. Intelligent Streaming Configuration
  4. Connections
  5. Sources and Targets in a Streaming Mapping
  6. Intelligent Streaming Mappings
  7. Window Transformation
  8. Data Type Reference

Big Data Streaming User Guide

Big Data Streaming User Guide

Transformations in a Streaming Mapping

Transformations in a Streaming Mapping

Informatica Developer provides a set of transformations that perform specific functions. Some restrictions and guidelines apply to processing transformations in a Streaming mapping.
The following table describes rules and guidelines for the transformations that are supported in a Streaming mapping:
Transformation
Rules and Guidelines
Aggregator
Mapping validation fails in the following situations:
  • The transformation contains stateful variable ports.
  • The transformation contains unsupported functions in an expression.
Expression
Mapping validation fails in the following situations:
  • The transformation contains stateful variable ports.
  • The transformation contains unsupported functions in an expression.
If an expression results in numerical errors, such as division by zero or SQRT of a negative number, it returns an infinite or an NaN value. In the native environment, the expression returns null values and the rows do not appear in the output.
Filter
Supported without restrictions.
Java
The following restrictions apply to the Java transformation:
  • The value Transaction for transformation scope is not valid.
  • The transformation is always Stateless
  • The Partitionable field is ignored.
Joiner
Mapping validation fails in the following situations:
  • Case sensitivity is disabled.
Lookup
Use a Lookup transformation to look up data in a flat file, HDFS, Hive, or Sqoop.
Mapping validation fails in the following situations:
  • Case sensitivity is disabled.
  • The lookup is a data object.
  • The cache is configured to be shared, named, persistent, dynamic, or uncached. The cache must be a static cache.
The mapping fails in the following situations:
  • The transformation is unconnected.
You cannot use a float data type to look up data in a Hive table as comparing equality of floating point numbers is unsafe.
When you configure the transformation to return the first, last, or any value on multiple matches, the Data Integration Service returns any value.
To use a Lookup transformation on Sqoop in a Cloudera distribution, perform the following configuration:
  1. In the Yarn configuration, locate the property
    NodeManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml
  2. Add the following xml snippet:
    <property> <name>mapreduce.application.classpath</name> <value>$HADOOP_MAPRED_HOME/,$HADOOP_MAPRED_HOME/lib/, $MR2_CLASSPATH</value> </property>
Informatica recommends that you select the
Ignore null values that match
property in Lookup transformation advanced properties to avoid cross join of DataFrames.
Rank
Supports only case sensitive string comparison.
Router
Supported without restrictions.
Sorter
To use the Sorter transformation in a Streaming mapping, configure the following properties:
  • Advanced properties of the data object write properties. Enable the
    Maintain Row Order
    field.
  • Custom properties of the Data Integration Service. Set the
    ExecutionContextOptions.Infa.HonorTargetOrdering
    property to true if there are one or more transformations between the Sorter transformation and the target.
Mapping validation fails in the following situations:
  • Case sensitivity is disabled.
The Data Integration Service logs a warning and ignores the Sorter transformation in the following situations:
  • There is a type mismatch in between the target and the Sorter transformation sort keys.
  • The transformation contains sort keys that are not connected to the target.
  • The transformation is not directly upstream from the Write transformation.
The Data Integration Service treats null values as high even if you configure the transformation to treat null values as low.
Union
Supported without restrictions.
Window
Supported without restrictions.
See the Window Transformation chapter in this guide for more information.
Transformations not listed in this table are not supported.
For more information about the transformations, see the
Informatica Developer Transformation Guide
.
For more information about restrictions on the Spark engine, see the
Informatica Big Data Management User Guide
.