Table of Contents

Search

  1. Preface
  2. Understanding Pipeline Partitioning
  3. Partition Points
  4. Partition Types
  5. Pushdown Optimization
  6. Pushdown Optimization and Transformations
  7. Real-time Processing
  8. Commit Points
  9. Row Error Logging
  10. Workflow Recovery
  11. Stopping and Aborting
  12. Concurrent Workflows
  13. Grid Processing
  14. Load Balancer
  15. Workflow Variables
  16. Parameters and Variables in Sessions
  17. Parameter Files
  18. FastExport
  19. External Loading
  20. FTP
  21. Session Caches
  22. Incremental Aggregation
  23. Session Log Interface
  24. Understanding Buffer Memory
  25. High Precision Data

Advanced Workflow Guide

Advanced Workflow Guide

Using Sorter Transformations

Using Sorter Transformations

If the session uses a Sorter transformation to sort data, use
n
:
n
partitions for the master and detail pipelines. Use a hash auto-keys partition point at the Sorter transformation to group the data. You can add a pass-through or hash auto-keys partition point at the Joiner transformation.
The Integration Service groups data into partitions of the same hash values, and the Sorter transformation sorts the data before passing it to the Joiner transformation. When the Integration Service processes the Joiner transformation configured with a hash auto-keys partition, it maintains the sort order by processing the sorted data with the same partitions it uses to route the data from each Sorter transformation.
The following image shows Sorter transformations used with hash auto-keys partitions to maintain sort order:
 The master and detail pipelines contain a source with unsorted data. Each Source Qualifier transformation links to a Sorter transformation. Each Sorter transformation has a hash-auto keys partition point. The Sorter transformations send sorted data to a Joiner transformation. The Joiner transformation can have either a hash-auto keys partition point or a pass-through partition point. The Joiner transformation sends sorted and joined data downstream.
For best performance, use sorted flat files or sorted relational data. You might want to calculate the processing overhead for adding Sorter transformations to the mapping.


Updated June 25, 2020