Table of Contents

Search

  1. Preface
  2. Understanding Pipeline Partitioning
  3. Partition Points
  4. Partition Types
  5. Pushdown Optimization
  6. Pushdown Optimization and Transformations
  7. Real-time Processing
  8. Commit Points
  9. Row Error Logging
  10. Workflow Recovery
  11. Stopping and Aborting
  12. Concurrent Workflows
  13. Grid Processing
  14. Load Balancer
  15. Workflow Variables
  16. Parameters and Variables in Sessions
  17. Parameter Files
  18. FastExport
  19. External Loading
  20. FTP
  21. Session Caches
  22. Incremental Aggregation
  23. Session Log Interface
  24. Understanding Buffer Memory
  25. High Precision Data

Advanced Workflow Guide

Advanced Workflow Guide

Rules and Guidelines for Adding and Deleting Partition Points

Rules and Guidelines for Adding and Deleting Partition Points

The following rules and guidelines apply when adding and deleting partition points:
  • You cannot create a partition point at a source instance.
  • You cannot create a partition point at a Sequence Generator transformation or an unconnected transformation.
  • You can add a partition point at any other transformation provided that no partition point receives input from more than one pipeline stage.
  • You cannot delete a partition point at a Source Qualifier transformation, a Normalizer transformation for COBOL sources, or a target instance.
  • You cannot delete a partition point at a multiple input group Custom transformation that is configured to use one thread per partition.
  • You cannot delete a partition point at a multiple input group transformation that is upstream from a multiple input group Custom transformation that is configured to use one thread per partition.
  • The following partition types have restrictions with dynamic partitioning:
    • Pass-through. When you use dynamic partitioning, if you change the number of partitions at a partition point, the number of partitions in each pipeline stage changes.
    • Key range. To use key range with dynamic partitioning you must define a closed range of numbers or date keys. If you use an open-ended range, the session runs with one partition.
You can add and delete partition points at other transformations in the pipeline according to the following rules:
  • You cannot create partition points at source instances.
  • You cannot create partition points at Sequence Generator transformations or unconnected transformations.
  • You can add partition points at any other transformation provided that no partition point receives input from more than one pipeline stage.
The following figure shows the valid partition points in a mapping:
In this mapping, a source connects to a Source Qualifier transformation, which connects to two Expression transformations. The Expression transformations and a Sequence Generator transformation connect to a third Expression transformation, which connects to a target.
In this mapping, the Workflow Manager creates partition points at the source qualifier and target instance by default. You can place an additional partition point at Expression transformation EXP_3.
If you place a partition point at EXP_3 and define one partition, the master thread creates the following threads:
A partition point exists at SQ_Source, EXP_3, and the target. A new thread begins at each partition point.
  1. Reader Thread.
  2. Transformation Threads.
  3. Writer Thread.
In this case, each partition point receives data from only one pipeline stage, so EXP_3 is a valid partition point.
The following transformations are not valid partition points:
Transformation
Reason
Source
Source instance.
SG_1
Sequence Generator transformation.
EXP_1 and EXP_2
If you could place a partition point at EXP_1 or EXP_2, you would create an additional pipeline stage that processes data from the source qualifier to EXP_1 or EXP_2. In this case, EXP_3 would receive data from two pipeline stages, which is not allowed.


Updated June 03, 2019