Table of Contents

Search

  1. Preface
  2. Understanding Pipeline Partitioning
  3. Partition Points
  4. Partition Types
  5. Pushdown Optimization
  6. Pushdown Optimization and Transformations
  7. Real-time Processing
  8. Commit Points
  9. Row Error Logging
  10. Workflow Recovery
  11. Stopping and Aborting
  12. Concurrent Workflows
  13. Grid Processing
  14. Load Balancer
  15. Workflow Variables
  16. Parameters and Variables in Sessions
  17. Parameter Files
  18. FastExport
  19. External Loading
  20. FTP
  21. Session Caches
  22. Incremental Aggregation
  23. Session Log Interface
  24. Understanding Buffer Memory
  25. High Precision Data

Advanced Workflow Guide

Advanced Workflow Guide

Using 1:n Partitions

Using 1:n Partitions

If the session uses sorted relational data, use one partition for the master source and
n
partitions for the detail source (1:
n
). Add a key-range or pass-through partition point at the Source Qualifier transformation. Do not add a partition point at the Joiner transformation. The Integration Service maintains the sort order when you create one partition for the master source because it does not redistribute data among partitions.
When you sort relational data with 1:
n
partitioning, the Joiner transformation might output unsorted data based on the join type. If you use a full outer or detail outer join, the Integration Service processes unmatched master rows last, which can result in unsorted data.
The following image shows sorted relational data with 1:n partitioning:
 The master pipeline in the mapping contains one unsorted relational source, and the detail pipeline contains multiple unsorted relational sources. The Source Qualifier transformation in the detail pipeline can use either a key-range or a pass-through partition point. Both pipelines send sorted data to a Joiner transformation. The Joiner transformation can output sorted data, based on the join type.


Updated June 25, 2020