Performance Tuning and Sizing Guidelines for PowerExchange for Amazon Redshift on the Spark Engine

Back Next

Troubleshooting Spark Tuning Parameters

Use the following section to troubleshoot errors while tuning the Spark parameters:

ERROR: "Container is running beyond physical memory limits in EMR cluster": To resolve this error, increase the
spark.executor.memory
parameter to 2 GB for a small, medium, or large data sets.
ERROR: "java.lang.IllegalArgumentException: Required executor memory (6144+614 MB) is above the max threshold (6144 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'": To resolve this error, increase the
yarn.scheduler.maximum-allocation-mb
parameter to 10 GB or decrease the
spark.executor.memory
parameter from 6 GB to 4 GB for a default data set.