Performance tuning and best practices for Google BigQuery V2 Connector

Performance tuning and best practices for Google BigQuery V2 Connector

Configure Secure Agent concurrency

Configure Secure Agent concurrency

By default, the Secure Agent processes two concurrent mapping tasks.
If there are more than two concurrent tasks, additional tasks are queued and then scheduled for execution when a slot becomes available. This can cause the capacity of the Secure Agent machine to be underutilized.
To achieve better utilization of the CPU capacity of the Secure Agent machine and achieve a higher degree of concurrency, you can set the
maxDTMProcesses
custom property for the Data Integration Server to the number of parallel tasks.
To configure the Secure Agent concurrency, perform the following steps:
  1. In Administrator, open the
    Runtime Environments
    page.
  2. Select the Secure Agent to view its details, and then click Edit on the Secure Agent page.
  3. Specify the following details:
    • Service. Choose
      Data_Integration_Server
      .
    • Type. Choose
      Tomcat
      .
    • Name. Enter
      maxDTMProcesses
      .
    • Value. Enter the Secure Agent concurrency value.
  4. Click Save.
For example, if you set the maxDTMProcesses property to 16, the Secure Agent allows 16 mapping tasks to run simultaneously.
Each mapping task spawns its own JVM and reserves the specified JVM heap from the physical memory.