Databricks Connector

Back Next

Configure Spark parameters

Before you connect to the job cluster, you must configure the Spark parameters on AWS and Azure.

Add the following Spark configuration parameters for the job cluster and restart the cluster:

spark.hadoop.fs.s3a.access.key <value>

spark.hadoop.fs.s3a.secret.key <value>

spark.hadoop.fs.s3a.endpoint <value>

Ensure that the access and secret key configured has access to the buckets where you store the data for Databricks tables.

Add the following Spark configuration parameters for the job cluster and restart the cluster:

fs.azure.account.oauth2.client.id.<storage-account-name>.dfs.core.windows.net <value>

fs.azure.account.auth.type.<storage-account-name>.dfs.core.windows.net <value>

fs.azure.account.oauth2.client.secret.<storage-account-name>.dfs.core.windows.net <Value>

fs.azure.account.oauth.provider.type.<storage-account-name>.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider

fs.azure.account.oauth2.client.endpoint.<storage-account-name>.dfs.core.windows.net https://login.microsoftonline.com/<Tenant ID>/oauth2/token

Ensure that the client ID and client secret configured has access to the file systems where you store the data for Databricks tables.