Table of Contents

Search

  1. Preface
  2. Introduction to Big Data Streaming
  3. Big Data Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Connections
  9. Sample Files

Rules and Guidelines for the Window Transformations

Rules and Guidelines for the Window Transformations

Certain transformations are valid with restrictions with the Window transformation. The following table describes the rules and guidelines for transformations:
Transformation
Rules and Guidelines
Aggregator
The Aggregator transformation is a multi-group active transformation.
The following rules apply to Aggregator transformations:
  • You must use a Window transformation directly upstream from an Aggregator transformation in a streaming mapping.
  • You cannot use multiple Aggregator transformations in the same streaming pipeline.
  • If you connect the Window port from a Window transformation to an Aggregator transformation, you cannot connect the Window port to any downstream transformation.
  • If a mapplet contains an Aggregator transformation, you must include a Window transformation directly upstream from the mapplet.
  • You cannot perform the group by aggregations on date/time data type port marked as a Window port. If you want to perform aggregations on the date/time data type port, you must create a date/time data type port with the timestamp values, and then perform the group by aggregations on the newly created data type port.
Joiner
The Joiner transformation is a multi-group active transformation.
The following rules apply to Joiner transformations:
  • You must use a Window transformation directly upstream from a Joiner transformation in a streaming mapping.
  • You cannot use multiple Joiner transformations in the same streaming pipeline.
  • You must use a Window transformation between the streaming source and any Joiner transformation in a streaming mapping.
  • The upstream Window transformations in pipelines to a Joiner transformation must have the same slide intervals.
  • You cannot use an Aggregator tranformation anywhere before a Joiner transformation in a streaming mapping.
  • If you connect the Window port from a Window transformation to a Joiner transformation, you cannot connect the Window port to any downstream transformation.
  • If a mapplet contains a Joiner transformation, you must include a Window transformation directly upstream from the mapplet.
Lookup
The following rules apply to Lookup transformations:
  • A Lookup transformation does not require a Window transformation between a streaming source and itself.
  • You can include a Lookup transformation only if the mapping has flat file, Hive, HBase, HDFS or JDBC sources.
  • You cannot use an Aggregator transformation and a passive Lookup transformation that is configured with an inequality lookup condition in the same pipeline.
  • You cannot use more than one passive Lookup transformation that is configured with an inequality lookup condition in the same pipeline.
Rank
You must use a Window transformation before a Rank transformation in a streaming mapping.
Support for the Rank transformation is deferred. Support will be reinstated
Sorter
The Sorter transformation is an active transformation. You must use a Window transformation between the streaming source and the Sorter transformation in a streaming mapping.
Support for the Sorter transformation is deferred. Support will be reinstated in a future release.
Union
The following rules apply to Union transformations:
  • A Union transformation does not require a Window transformation between a streaming source and itself.
  • A Union transformation cannot be used to merge data from streaming and non-streaming pipelines.
Window
The following rules apply to Window transformations:
  • You cannot add a Window transformation to a Logical Data Object mapping or mapplet.
  • A Window transformation must have at least one upstream streaming source.
  • All Window transformations must have a slide interval that is a multiple of the mapping batch interval.
  • A Window transformation that is downstream from another Window transformation must have a slide interval that is a multiple of the slide interval of the upstream Window transformation.
  • The slide interval of a sliding Window transformation must be less than window size.
  • The format of the parameter of the window size must have the TimeDuration parameter type.
  • The window size and the slide interval of a Window transformation must be greater than 0.
  • The window port from the Window transformation cannot be connected to more than one downstream transformation.