Table of Contents

Search

  1. Preface
  2. Understanding Pipeline Partitioning
  3. Partition Points
  4. Partition Types
  5. Pushdown Optimization
  6. Pushdown Optimization and Transformations
  7. Real-time Processing
  8. Commit Points
  9. Row Error Logging
  10. Workflow Recovery
  11. Stopping and Aborting
  12. Concurrent Workflows
  13. Grid Processing
  14. Load Balancer
  15. Workflow Variables
  16. Parameters and Variables in Sessions
  17. Parameter Files
  18. FastExport
  19. External Loading
  20. FTP
  21. Session Caches
  22. Incremental Aggregation
  23. Session Log Interface
  24. Understanding Buffer Memory
  25. High Precision Data

Advanced Workflow Guide

Advanced Workflow Guide

Configuring Sessions to Use Multiple Threads

Configuring Sessions to Use Multiple Threads

To configure a session to read a file with multiple threads, leave the source file name blank for partitions 2-
n
. The Integration Service uses partitions 2-
n
to read a portion of the previous partition file or file list. The Integration Service ignores the directory field of that partition.
To configure a session to read from a command with multiple threads, enter a command for each partition or leave the command property blank for partitions 2-
n
. If you enter a command for each partition, the Integration Service creates a thread to read the data generated by each command. Otherwise, the Integration Service uses partitions 2-
n
to read a portion of the data generated by the command for the first partition.
The following table shows the attributes and values when the Integration Service creates three threads to concurrently read ProductsA.txt:
Attribute
Value
Partition #1
ProductsA.txt
Partition #2
<blank>
Partition #3
<blank>
The following table shows the attributes and values when the Integration Service creates three threads to read ProductsA.txt and ProductsB.txt concurrently. Two threads read ProductsA.txt and one thread reads ProductsB.txt:
Attribute
Value
Partition #1
ProductsA.txt
Partition #2
<blank>
Partition #3
ProductsB.txt
The following table shows the attributes and values when the Integration Service creates three threads to concurrently read data piped from the command:
Attribute
Value
Partition #1
CommandA
Partition #2
<blank>
Partition #3
<blank>
The following table shows the attributes and values when the Integration Service creates three threads to read data piped from CommandA and CommandB. Two threads read the data piped from CommandA and one thread reads the data piped from CommandB:
Attribute
Value
Partition #1
CommandA
Partition #2
<blank>
Partition #3
CommandB


Updated August 27, 2020