Performance Tuning Guide

10.5.6
- 10.5
- 10.4.0

Back Next

Increasing the Number of Partitions

You can increase the number of partitions in a pipeline to improve session performance. Increasing the number of partitions allows the Integration Service to create multiple connections to sources and process partitions of source data concurrently.

When a session uses a file source, you can configure it to read the source with one thread or multiple threads. Configure the session to read file sources with multiple threads to increase session performance. The Integration Service creates multiple concurrent connections to the file source.

When you create a session, the Workflow Manager validates each pipeline in the mapping for partitioning. You can specify multiple partitions in a pipeline if the Integration Service can maintain data consistency when it processes the partitioned data.

Use the following tips when you add partitions to a session:

Add one partition at a time.
To best monitor performance, add one partition at a time, and note the session settings before you add each partition.

Set DTM Buffer Memory.
When you increase the number of partitions, increase the DTM buffer size. If the session contains n
partitions, increase the DTM buffer size to at least n times the value for the session with one partition.

Set cached values for Sequence Generator.
If a session has n
partitions, you should not need to use the “Number of Cached Values” property for the Sequence Generator transformation. If you set this value to a value greater than 0, make sure it is at least n
times the original value for the session with one partition.

Partition the source data evenly.
Configure each partition to extract the same number of rows.

Monitor the system while running the session.
If CPU cycles are available, you can add a partition to improve performance. For example, you may have CPU cycles available if the system has 20 percent idle time.

Monitor the system after adding a partition.
If the CPU utilization does not go up, the wait for I/O time goes up, or the total data transformation rate goes down, then there is probably a hardware or software bottleneck. If the wait for I/O time goes up by a significant amount, then check the system for hardware bottlenecks. Otherwise, check the database configuration.

Rename Saved Search

Table of Contents

Performance Tuning Guide

Performance Tuning Guide

Increasing the Number of Partitions

Increasing the Number of Partitions