Table of Contents

Search

  1. Preface
  2. Introduction to Big Data Streaming
  3. Big Data Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Connections
  9. Sample Files

Transformations in a Streaming Mapping

Transformations in a Streaming Mapping

Informatica Developer provides a set of transformations that perform specific functions. Some restrictions and guidelines apply to processing transformations in a streaming mapping.
You can use the following transformations in a streaming mapping:
Aggregator
Mapping validation fails when the
  • The transformation contains stateful variable ports.
  • The transformation contains unsupported functions in an expression.
Data Masking
Supported without restrictions.
Support for the Data Masking transformation is deferred. Support will be reinstated in a future release.
Expression
Mapping validation fails when the transformation contains unsupported functions in an expression.
If an expression results in numerical errors, such as division by zero or SQRT of a negative number, it returns an infinite or an NaN value. In the native environment, the expression returns null values and the rows do not appear in the output.
Filter
Supported without restrictions.
Java
The following restrictions apply to the Java transformation:
  • The value Transaction for transformation scope is not valid.
  • The transformation is always stateless.
  • The Partitionable field is ignored.
You can use all data types to process hierarchical data.
For more information about Java transformation support on the Spark engine, see the
Informatica Big Data Management User Guide
.
Joiner
Mapping validation fails in the following situations:
  • Case sensitivity is disabled.
Lookup
Use a Lookup transformation to look up data in a flat file, HDFS, Hive, relational, and HBase data.
Mapping validation fails in the following situations:
  • Case sensitivity is disabled.
  • The lookup is a data object.
The mapping fails in the following situations:
  • The transformation is unconnected.
You cannot use a float data type to look up data in a Hive table as comparing equality of floating point numbers is unsafe.
When you configure the transformation to return the first, last, or any value on multiple matches, the Data Integration Service returns any value.
Informatica recommends that you select the
Ignore null values that match
property in Lookup transformation advanced properties to avoid cross join of DataFrames.
To use a Lookup transformation on uncached HBase tables, perform the following steps:
  1. Create an HBase data object. When you add an HBase table as the resource for a HBase data object, include the ROW ID column.
  2. Create a HBase read data operation and import it into the streaming mapping.
  3. When you import the data operation to the mapping, select the
    Lookup
    option.
  4. In the Lookup tab, configure the following options:
    • Lookup column. Specify an equality condition on ROW ID
    • Operator. Specify =
  5. Verify that format for any date value in the HBase tables is of a valid Java date format. Specify this format in the
    Date Time Format
    property of the
    Advanced Properties
    tab of the data object read operation.
If an HBase lookup does not result in a match, it generates a row with null values for all columns. You can add a Filter transformation after the Lookup transformation to filter out null rows.
Mapping validation fails in the following situations:
  • If you do not include ROW ID in the condition
  • If you specify any operator other than =
  • If you include multiple conditions in the transformation.
  • If you select column of type date from input columns.
  • If you look up binary data.
Normalizer
Supported without restrictions.
Python
When you close a Jep instance, you might not be able to call CPython modules.
Rank
Mapping validation fails if case-sensitivity is disabled.
Support for the Rank transformation is deferred. Support will be reinstated in a future release.
Router
Supported without restrictions.
Sorter
To use the Sorter transformation in a Streaming mapping, configure the following properties:
  • Advanced properties of the data object write properties. Enable the
    Maintain Row Order
    field.
  • Custom properties of the Data Integration Service. Set the
    ExecutionContextOptions.Infa.HonorTargetOrdering
    property to true if there are one or more transformations between the Sorter transformation and the target.
Mapping validation fails in the following situations:
  • Case sensitivity is disabled.
The Data Integration Service logs a warning and ignores the Sorter transformation in the following situations:
  • There is a type mismatch in between the target and the Sorter transformation sort keys.
  • The transformation contains sort keys that are not connected to the target.
  • The transformation is not directly upstream from the Write transformation.
The Data Integration Service treats null values as high even if you configure the transformation to treat null values as low.
Support for the Sorter transformation is deferred. Support will be reinstated in a future release.
Union
Supported without restrictions.
Window
The Window transformation is supported with restrictions. The Window transformation applies only to streaming mappings.
For more information about the restrictions, see the Rules and Guidelines for the Window Transformations.
Transformations that are not listed here are not supported.
For more information about the transformations, see the
Informatica Developer Transformation Guide
.
For more information about transformation support on the Spark engine, see the
Informatica Big Data Management User Guide
.