Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Data Engineering Integration
  3. Mappings
  4. Mapping Optimization
  5. Sources
  6. Targets
  7. Transformations
  8. Python Transformation
  9. Data Preview
  10. Cluster Workflows
  11. Profiles
  12. Monitoring
  13. Hierarchical Data Processing
  14. Hierarchical Data Processing Configuration
  15. Hierarchical Data Processing with Schema Changes
  16. Intelligent Structure Models
  17. Blockchain
  18. Stateful Computing
  19. Appendix A: Connections Reference
  20. Appendix B: Data Type Reference
  21. Appendix C: Function Reference

Configure Databricks Clusters Using Warm Pools

Configure Databricks Clusters Using Warm Pools

On the Azure and AWS Databricks platforms, you can configure Databricks ephemeral clusters to start on sets of cached VM instances, called warm pools. The VM instances wait on standby in a running state for ephemeral cluster creation. When you enable warm pools, you reduce the time it takes to create clusters if the pool contains enough idle instances.

Prerequisite

Ephemeral clusters from warm pools require the Databricks administrator to create a cluster pool in the Databricks workspace. Choose from any available instance types for the VMs in the pool.
Configure the following properties when you create pools:
Property
Description
Min Idle
Minimum number of instances that will remain running in the pool in an idle state. These instances do not terminate, regardless of the setting specified in the Idle Instance Auto Termination property.
Idle Instance Auto Termination
The number of minutes that the pool will maintain an instance in an idle state after jobs are complete.
The idle instances in the pool do not terminate regardless of the value in the Idle Instance Auto Termination property. Amazon charges for idle instances in a pool.

Configure the Cluster Workflow to Use Warm Pool Instances

Enable warm pools for Databricks ephemeral clusters in the Create Cluster task Advanced Properties.
The following table describes the properties to enable warm pools:
Property
Description
Warm Pools
Check this option to enable creation of the Databricks cluster using warm pools.
Default is unchecked (false).
Pool ID
Required when you enable warm pools. Value of the DatabricksInstancePoolId property that identifies the warm pool resource.
When you enable warm pools, some of the advanced properties that allow you to specify cluster instance details are disabled. Set cluster instance details when you configure pools in the Databricks console.


Updated March 31, 2021