Performance tuning and best practices for Google BigQuery V2 Connector

Performance tuning and best practices for Google BigQuery V2 Connector

Bulk processing for write operations

Bulk processing for write operations

You can enable bulk processing to write large amounts of data to Google BigQuery. Bulk processing utilizes minimal number of API calls and the performance of the write operation is optimized.
To enable bulk processing, specify the property
-DENABLE_WRITER_BULK_PROCESSING=true
in the Secure Agent properties:
Perform the following steps to configure bulk processing before you run a mapping:
  1. In Administrator, select the Secure Agent listed on the
    Runtime Environments
    tab.
  2. Click
    Edit
    .
  3. In the
    System Configuration Details
    section, select
    Data Integration Server
    as the service and
    DTM
    as the type.
  4. Edit the JVM option, and enter
    -DENABLE_WRITER_BULK_PROCESSING=true
    .
  5. Click
    Save
    .
The following graph illustrates the impact of partitioning on the writer performance:
The performance increases when you configure bulk processing to write data to Google BigQuery.
This feature does not apply for mapping tasks configured with pushdown optimization.

0 COMMENTS

We’d like to hear from you!