Amazon Redshift Connector Best Practices

Back Next

Transformation Tuning

When you initialize a mapping, blocks of memory are allocated to hold the source and target data. If enough memory blocks to hold the data are not allocated, the mapping fails.

The DTM buffer is a temporary storage area used to store temporary data for caching data used by transformation. The buffer is divided into blocks. The buffer size and block size for a mapping are tunable. If the performance of read, write, or any transformations are slow, this is an indication that DTM buffer tuning is required.

You must tune the DTM buffer size and buffer block size for optimal throughput in the following manner:

Default Buffer Block Size: To adjust the buffer block size, use the following code:
Buffer Block Size = sum of all column widths (in bytes) * n, where n is a positive integer > 1

In the advanced session property, select the
Default buffer block size
option and set the required value. The following image shows the
Default buffer block size
advanced session property:
DTM Buffer Size: To adjust the DTM buffer size, use the following code:
DTM Buffer Size = Buffer Block size * m, Where m is a positive integer >1

In the advanced session property, select the
DTM Buffer Size
advanced session propertyoption and set the required value. The following image shows the
DTM Buffer Size
advanced session property:

The DTM buffer size is based on the size of the row and data. If data partitioning is enabled, the DTM buffer size is the total size of all memory buffer allocated to all partitions. For a mapping that contains

numbers of partitions, set the value of the

DTM Buffer Size

option to at least

times for the mapping with one partition.

Amazon Redshift Connector Best Practices

Download Guide

Watch

Comments

Communities

Knowledge Base

Success Portal