Table of Contents

Search

  1. Preface
  2. Introduction to Big Data Streaming
  3. Big Data Streaming Configuration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Appendix A: Connections
  9. Appendix B: Data Type Reference
  10. Appendix C: Sample Files

Big Data Streaming User Guide

Big Data Streaming User Guide

Rules and Guidelines for Transformations

Rules and Guidelines for Transformations

Certain transformations are valid with restrictions with the Window transformation. The following table describes the rules and guidelines for transformations:
Transformation
Rules and Guidelines
Aggregator
The Aggregator transformation is a multiple-group active transformation.
You must use a Window transformation before an Aggregator transformation in a Streaming mapping.
If you do not specify the group by ports to define groups for aggregations, the transformation sends a value of 0 when it receives no data.
Joiner
The Joiner transformation is a multiple-group active transformation.
The following rules apply to Joiner transformations:
  • You must use a Window transformation between the streaming source and any Joiner transformation in a Streaming mapping.
  • The upstream Window transformations in pipelines to a Joiner transformation must have the same slide intervals.
Lookup
The following rules apply to Lookup transformations:
  • A Lookup transformation does not require a Window transformation between a streaming source and itself.
  • You can include a Lookup transformation only if the mapping has flat file, Hive, HBase, or JDBC sources.
Rank
You must use a Window transformation before a Rank transformation in a Streaming mapping.
Sorter
The Sorter transformation is a multiple-group active transformation. You must use a Window transformation between the streaming source and the Sorter transformation in a Streaming mapping.
Union
The following rules apply to Union transformations:
  • A Union transformation does not require a Window transformation between a streaming source and itself.
  • If one pipeline to a Union transformation has a Window transformation, all streaming pipelines must have a Window transformation. All upstream Window transformations in the pipelines to the Union transformations must have the same slide interval.
  • A Union transformation cannot be used to merge data from streaming and non-streaming pipelines.
Window
The following rules apply to Window transformations:
  • You cannot add a Window transformation to an Logical Data Object mapping, REST mapping, or mapplet.
  • A Window transformation must have at least one upstream streaming source.
  • All Window transformations must have a slide interval that is a multiple of the mapping batch interval.
  • A Window transformation that is downstream from another Window transformation must have a slide interval that is a multiple of the slide interval of the upstream Window transformation.
  • The slide interval of a sliding Window transformation must be less than window size.
  • The format of the parameter of the window size must have the TimeDuration parameter type.
  • The window size and slide interval of a Window transformation must be greater than 0.

0 COMMENTS

We’d like to hear from you!