Table of Contents

Search

  1. Preface
  2. Understanding Pipeline Partitioning
  3. Partition Points
  4. Partition Types
  5. Pushdown Optimization
  6. Pushdown Optimization and Transformations
  7. Real-time Processing
  8. Commit Points
  9. Row Error Logging
  10. Workflow Recovery
  11. Stopping and Aborting
  12. Concurrent Workflows
  13. Grid Processing
  14. Load Balancer
  15. Workflow Variables
  16. Parameters and Variables in Sessions
  17. Parameter Files
  18. FastExport
  19. External Loading
  20. FTP
  21. Session Caches
  22. Incremental Aggregation
  23. Session Log Interface
  24. Understanding Buffer Memory
  25. High Precision Data

Advanced Workflow Guide

Advanced Workflow Guide

Setting Partition Types

Setting Partition Types

The Workflow Manager sets a default partition type for each partition point in the pipeline. The Workflow Manager specifies pass-through as the default partition type for all partition points unless the transformation scope for a transformation is All Input. You can change the default type.
For example, at the source qualifier and target instance, the Workflow Manager specifies pass-through partitioning. For Rank and unsorted Aggregator transformations, the Workflow Manager specifies hash auto-keys partitioning when the transformation scope is All Input.
You must specify pass-through partitioning for all transformations that are downstream from a transaction generator or an active source that generates commits and upstream from a target or a transformation with Transaction transformation scope. Also, if you configure the session to use constraint-based loading, you must specify pass-through partitioning for all transformations that are downstream from the last active source.
If workflow recovery is enabled, the Workflow Manager sets the partition type to pass-through unless the partition point is either an Aggregator transformation or a Rank transformation.
You cannot create partition points for the following transformations:
  • Source definition
  • Sequence Generator
  • XML Parser
  • XML target
  • Unconnected transformations
The following table lists valid partition types and the default partition type for different partition points in the pipeline:
Transformation
(Partition Point)
Round- Robin
Hash Auto-Keys
Hash User Keys
Key Range
Pass-Through
Database
Partitioning
Source Qualifier
(relational sources)
no
no
no
yes
yes
yes
(Oracle, DB2)
Source Qualifier
(flat file sources)
no
no
no
no
yes
no
Web Service Source Qualifier
no
no
no
no
yes
no
XML Source Qualifier
no
no
no
no
yes
no
Normalizer
(COBOL sources)
no
no
no
no
yes
no
Normalizer
(relational)
yes
no
yes
yes
yes
no
Aggregator (sorted)
no
no
no
no
yes
no
Aggregator (unsorted)
no
yes
no
no
yes
no
Custom
yes
no
yes
yes
yes
no
Data Masking
yes
no
yes
yes
yes
no
Expression
yes
no
yes
yes
yes
no
External Procedure
yes
no
yes
yes
yes
no
Filter
yes
no
yes
yes
yes
no
HTTP
no
no
no
no
yes
no
Java
yes
no
yes
yes
yes
no
Joiner
no
yes
no
no
yes
no
Lookup
yes
yes
yes
yes
yes
no
Rank
no
yes
no
no
yes
no
Router
yes
no
yes
yes
yes
no
Sorter
no
yes
no
yes
yes
no
Stored Procedure
yes
no
yes
yes
yes
no
Transaction Control
yes
no
yes
yes
yes
no
Union
yes
no
yes
yes
yes
no
Unstructured Data
yes
no
yes
yes
yes
no
Update Strategy
yes
no
yes
yes
yes
no
Web Service Consumer
no
no
no
no
yes
no
XML Generator
no
no
no
no
yes
no
XML Parser
no
no
no
no
yes
no
Relational target definition
yes
no
yes
yes
yes
yes (DB2)
Flat file target definition
yes
no
yes
yes
yes
no
Web Service target
no
no
no
no
yes
no
For the following transformations, the default partition type is pass-through when the transformation scope is Transaction, and the default partition type is hash auto-keys when the transformation scope is All Input:
  • Aggregator (unsorted)
  • Joiner
  • Rank
  • Sorter


Updated June 25, 2020