tasks for execution. If there are more than two tasks, additional tasks are queued and then scheduled for execution when a slot becomes available. This can cause the Secure Agent machine's capacity to be underutilized.
To achieve better utilization of the CPU capacity of the Secure Agent machine and achieve a higher degree of concurrency, you can set the maxDTMProcesses custom property for the Data Integration Server to the number of parallel tasks. For example, setting this property to 16 allows 16 tasks to run simultaneously.
The following image shows configuration for the maxDTMProcesses custom property:
The recommended value for this property varies based on the connection types of the jobs that the agent runs:
If the jobs use file-based or ODBC connections, the recommended value for the Secure Agent machine is
0.75 * # logical CPUs
. For example, if there are 24 logical CPUs on the Secure Agent machine, set this value to 18.
If the jobs use cloud data warehouse or cloud data lake connectors, the recommended value for the Secure Agent machine is
0.33 * # logical CPUs
For example, if there are 24 logical CPUs on the Secure Agent machine, set this value to 8.
When you set the maxDTMProcesses custom property, note the following guidelines:
Setting this custom property to a value higher than the number of CPUs on the Secure Agent machine can help increase parallelism for task execution, but it can also cause performance bottlenecks in task execution time.
When you set this property, make sure that you do not exceed the terms of your license agreement. For example, if your organization is licensed for 3 Secure Agents and can run each agent on up to 4 CPUs, do not set this property to a value greater than 12.
ASK INFAPreview
