Abstract
Supported Versions
Sizing Guidelines and Performance Tuning for Big Data Streaming 10.2.1

Sizing Guidelines and Performance Tuning for Big Data Streaming 10.2.1

Back Next

Troubleshooting Spark Driver Errors

Spark driver uses significant amount of memory when it stores the execution details of complex mappings. In such cases, you might encounter an out of memory error similar to the following error:

ERROR MicroBatchExecution: Query Kafka_DM_Write3 [id = d0d2cabb-767d-4525-acc7-d9edefb4beee,

runId = 3f6865fa-1619-46a0-8413-18e9c6213e5d] terminated with error

java.lang.OutOfMemoryError: Java heap space

This error might occur if the

spark.sql.ui.retainedExecutions

property has a higher value. To avoid such an error, set the

spark.sql.ui.retainedExecutions

property to a lower value.

Recommendations for Tuning the Kafka Cluster

Download Guide

Watch

Comments

Communities

Knowledge Base

Success Portal