Advanced Clusters

Back Next

Resource requirements for cluster nodes

When you select instance types in an

advanced configuration

, make sure that the master and worker nodes have enough resources to run

advanced jobs

successfully.

Master node

The master node is recommended to have at least 8 GB of memory and 4 CPUs.

Because processing on the master node is network-intensive, avoid T instance types in an AWS environment.

Worker nodes

Worker nodes are recommended to have at least 16 GB of memory and 8 CPUs.

The following table lists the default resource requirements for worker nodes:

Component	Default memory requirement	Default CPU requirement
Kubernetes system	1 GB per worker node	0.5 CPU per worker node with an additional 0.5 CPU across the cluster
Spark shuffle service	2 GB per worker node	1 CPU per worker node
Spark driver	4 GB	0.75 CPU
Spark executor	6 GB, or 3 GB per Spark executor core	1.5 CPUs, or 0.75 CPU per Spark executor core

Based on the default resource requirements, a cluster with one worker node requires 13 GB of memory and 4.25 CPUs.

When worker nodes are added to the cluster, each worker node reserves an additional 3 GB of memory and 1.5 CPU for the Kubernetes system and the Spark shuffle service. Therefore, a cluster with two worker nodes requires 16 GB of memory and 5.75 CPUs.

Advanced configurations

Reconfiguring resource requirements

Resource requirements example

Download Guide

Watch

Comments

Communities

Knowledge Base

Success Portal