Advanced Clusters

Back Next

Reconfiguring resource requirements

If you cannot provision enough resources to fulfill the default requirements, you can reconfigure some of the requirements.

You can reconfigure the requirements for the following components:

Spark shuffle service: If you disable the shuffle service, the
Spark engine
cannot use dynamic allocation. For more details, contact Informatica Global Customer Support.
Spark driver: To reconfigure the amount of memory for the Spark driver, use the Spark session property
spark.driver.memory
in the
mapping
task. To set the memory in terms of GB, use a value such as 2G. To set the memory in terms of MB, use a value such as 1500m.

For information about reconfiguring the CPU requirement for the Spark driver, contact Informatica Global Customer Support.
Spark executor: To reconfigure the amount of memory for the Spark executor, use the Spark session property
spark.executor.memory
in the
mapping
task. Similar to the memory value for the Spark driver, you can specify the memory in GB or MB.
You can also change the number of Spark executor cores using the Spark session property
spark.executor.cores
. The default number of cores for GPU-enabled clusters is 4. The default number of cores for all other clusters is 2.

If you edit the number of cores, you change the number of Spark tasks that run concurrently. For example, two Spark tasks can run concurrently inside each Spark executor when you set spark.executor.cores=2.

For information about reconfiguring the CPU requirement for Spark executors, contact Informatica Global Customer Support.