Table of Contents

Search

  1. Preface
  2. Understanding Pipeline Partitioning
  3. Partition Points
  4. Partition Types
  5. Pushdown Optimization
  6. Pushdown Optimization and Transformations
  7. Real-time Processing
  8. Commit Points
  9. Row Error Logging
  10. Workflow Recovery
  11. Stopping and Aborting
  12. Concurrent Workflows
  13. Grid Processing
  14. Load Balancer
  15. Workflow Variables
  16. Parameters and Variables in Sessions
  17. Parameter Files
  18. FastExport
  19. External Loading
  20. FTP
  21. Session Caches
  22. Incremental Aggregation
  23. Session Log Interface
  24. Understanding Buffer Memory
  25. High Precision Data

Advanced Workflow Guide

Advanced Workflow Guide

Using n:n Partitions

Using n:n Partitions

If the session uses sorted relational data, use
n
:
n
partitions for the master and detail pipelines and add a pass-through or hash auto-keys partition point at the Joiner transformation.
When you use a pass-through partition at the Joiner transformation, maintain sorted data in the mapping. When you use a hash auto-keys partition point, you maintain the sort order by passing all sorted data to the Joiner transformation in a single partition. Add a key-range partition point at the Source Qualifier transformation that contains all source data in the first partition. When you pass sorted data in one partition, the Integration Service redistributes data among multiple partitions with a hash function and joins the sorted data.
The following image shows sorted relational data that pass through a single partition to maintain the sort order:
 The mapping has two pipelines. Each pipeline contains multiple relational sources with sorted data. Some of the sources in each pipeline have no data to send downstream. Each pipeline has a Source Qualifier transformation with a key-range partition point. Both pipelines link to one Joiner transformation. The Joiner transformation contains a hash-auto keys partition point, and it sends the sorted data downstream.


Updated June 25, 2020