Table of Contents


  1. Preface
  2. Advanced clusters
  3. Setting up AWS
  4. Setting up Google Cloud
  5. Setting up Microsoft Azure
  6. Setting up a self-service cluster
  7. Setting up a local cluster
  8. Advanced configurations
  9. Troubleshooting
  10. Appendix A: Command reference

Advanced Clusters

Advanced Clusters

Resource requirements for cluster nodes

Resource requirements for cluster nodes

When you select instance types in an
advanced configuration
, make sure that the master and worker nodes have enough resources to run
advanced jobs

Master node

The master node is recommended to have at least 8 GB of memory and 4 CPUs.
Because processing on the master node is network-intensive, avoid T instance types in an AWS environment.

Worker nodes

Worker nodes are recommended to have at least 16 GB of memory and 8 CPUs.
The following table lists the default resource requirements for worker nodes:
Default memory requirement
Default CPU requirement
Kubernetes system
1 GB per worker node
0.5 CPU per worker node with an additional 0.5 CPU across the cluster
Spark shuffle service
2 GB per worker node
1 CPU per worker node
Spark driver
4 GB
0.75 CPU
Spark executor
6 GB, or 3 GB per Spark executor core
1.5 CPUs, or 0.75 CPU per Spark executor core
Based on the default resource requirements, a cluster with one worker node requires 13 GB of memory and 4.25 CPUs.
When worker nodes are added to the cluster, each worker node reserves an additional 3 GB of memory and 1.5 CPU for the Kubernetes system and the Spark shuffle service. Therefore, a cluster with two worker nodes requires 16 GB of memory and 5.75 CPUs.


We’d like to hear from you!