Table of Contents

Search

  1. Preface
  2. Understanding Pipeline Partitioning
  3. Partition Points
  4. Partition Types
  5. Pushdown Optimization
  6. Pushdown Optimization and Transformations
  7. Real-time Processing
  8. Commit Points
  9. Row Error Logging
  10. Workflow Recovery
  11. Stopping and Aborting
  12. Concurrent Workflows
  13. Grid Processing
  14. Load Balancer
  15. Workflow Variables
  16. Parameters and Variables in Sessions
  17. Parameter Files
  18. FastExport
  19. External Loading
  20. FTP
  21. Session Caches
  22. Incremental Aggregation
  23. Session Log Interface
  24. Understanding Buffer Memory
  25. High Precision Data

Advanced Workflow Guide

Advanced Workflow Guide

Using n:n Partitions

Using n:n Partitions

If the session uses sorted flat file data, use
n
:
n
partitions for the master and detail pipelines. You can add a pass-through partition or hash auto-keys partition at the Joiner transformation.
If you add a pass-through partition at the Joiner transformation, maintain the sort order in mappings. If you add a hash auto-keys partition point at the Joiner transformation, you can maintain the sort order by passing all sorted data to the Joiner transformation in one partition. When you pass sorted data in one partition, the Integration Service maintains the sort order when it redistributes data with a hash function.
To allow the Integration Service to pass all sorted data in one partition, configure the session to use the sorted file for the first partition and empty files for the remaining partitions.
The Integration Service redistributes the rows among multiple partitions and joins the sorted data.
The following image shows sorted file data passed through one partition to maintain sort order:
The master and detail pipelines each contain multiple flat files. At least one flat file in each pipeline contains sorted data. Some flat files in each pipeline send no data downstream. Both pipelines link to a Source Qualifier transformation. Both pipelines link to a Joiner transformation. The Joiner transformation has a hash auto-keys partition point. The Joiner transformation sends sorted data downstream.


Updated November 14, 2019