Table of Contents

Search

  1. Preface
  2. Understanding Pipeline Partitioning
  3. Partition Points
  4. Partition Types
  5. Pushdown Optimization
  6. Pushdown Optimization and Transformations
  7. Real-time Processing
  8. Commit Points
  9. Row Error Logging
  10. Workflow Recovery
  11. Stopping and Aborting
  12. Concurrent Workflows
  13. Grid Processing
  14. Load Balancer
  15. Workflow Variables
  16. Parameters and Variables in Sessions
  17. Parameter Files
  18. FastExport
  19. External Loading
  20. FTP
  21. Session Caches
  22. Incremental Aggregation
  23. Session Log Interface
  24. Understanding Buffer Memory
  25. High Precision Data

Advanced Workflow Guide

Advanced Workflow Guide

Understanding Buffer Memory Overview

Understanding Buffer Memory Overview

When you run a session, the Integration Service process starts the Data Transformation Manager (DTM). The DTM allocates buffer memory to the session at run time based on the DTM Buffer Size setting in the session properties.
The DTM divides the memory into buffer blocks as configured in the Default Buffer Block Size setting in the session properties. The reader, transformation, and writer threads use buffer blocks to move data from sources to targets. The buffer block size should be larger than the precision for the largest row of data in a source or target.
The Integration Service allocates at least two buffer blocks for each source and target in a partition. For XML sources and targets, the buffer blocks must be at least twice the number of groups in the sources and targets. An XML reader with denormalized columns and XML schemas with circular references may require additional buffer blocks.
You configure buffer memory settings by adjusting the following session properties:
DTM Buffer Size
The DTM buffer size specifies the amount of buffer memory that the Integration Service uses when the DTM processes a session. Configure the DTM buffer size on the Properties tab in the session properties.
Default Buffer Block Size
The buffer block size specifies the amount of buffer memory used to move a block of data from the source to the target. Configure the buffer block size on the Config Object tab in the session properties.
The Integration Service calculates a minimum memory allocation for the buffer memory and buffer blocks. By default, the Integration Service allocates 64,000 bytes per block or the size of the largest row of any source or target in the mapping, whichever is larger.
If the DTM cannot allocate the configured amount of buffer memory for the session, the session cannot initialize. Usually, you do not need more than 1 GB for the buffer memory.
You can manually set a value for the buffer size, or you can configure the session to allow the Integration Service to determine the buffer memory size that the session requires.

0 COMMENTS

We’d like to hear from you!