Table of Contents

Search

  1. Preface
  2. Introduction to Data Engineering Administration
  3. Authentication
  4. Running Mappings on a Cluster with Kerberos Authentication
  5. Authorization
  6. Cluster Configuration
  7. Cloud Provisioning Configuration
  8. Data Integration Service Processing
  9. Appendix A: Connections Reference
  10. Appendix B: Monitoring REST API

Cluster Configuration Overview

Cluster Configuration Overview

A cluster configuration is an object in the domain that contains configuration information about the compute cluster. The cluster configuration enables the Data Integration Service to push mapping logic to the non-native environment. Import configuration properties from the compute cluster to create a cluster configuration. You can import directly from the cluster or from an archive file.
When you create a cluster configuration associated with the Hadoop environment, the
Cluster Configuration
wizard can create Hadoop, HBase, HDFS, and Hive connections to access the Hadoop environment. When you create a cluster configuration associated with the Databricks environment, the
Cluster Configuration
wizard can create the Databricks connection. If you choose to create the connections, the wizard also associates the configuration object with the connections.
You can override the property values, and you can create user-defined properties based on your requirements. When property values change on the cluster, you can refresh the cluster configuration, either directly from the cluster or from an archive file.
The cluster configuration contains properties for the cluster type and version. You can edit the version property to any supported version. After you change the version, you must restart the Data Integration Service.
Consider the following high-level process to manage cluster configurations:
  1. Import the cluster configuration, choosing to create connections that require the cluster configuration.
  2. Edit the cluster configuration. Override imported property values and add user-defined properties.
  3. Refresh the cluster configuration. When property values change on the cluster, refresh the cluster configuration to import the changes.

0 COMMENTS

We’d like to hear from you!