You can enable bulk processing to write large amounts of data to Google BigQuery. Bulk processing utilizes minimal number of API calls and the performance of the write operation is optimized.
To enable bulk processing, specify the property
-DENABLE_WRITER_BULK_PROCESSING=true
in the Secure Agent properties:
Perform the following steps to configure bulk processing before you run a mapping:
In Administrator, select the Secure Agent listed on the
Runtime Environments
tab.
Click
Edit
.
In the
System Configuration Details
section, select
Data Integration Server
as the service and
DTM
as the type.
Edit the JVM option, and enter
-DENABLE_WRITER_BULK_PROCESSING=true
.
Click
Save
.
The following graph illustrates the impact of partitioning on the writer performance:
This feature does not apply for mapping tasks configured with pushdown optimization.