Databricks Delta Connector

Back Next

Configure Spark parameters for AWS staging

Before you use the Databricks SQL warehouse to run mappings, configure the Spark parameters for SQL warehouse on the Databricks SQL Admin console.

On the Databricks SQL Admin console, navigate to

SQL Warehouse Settings > Data Security

, and then configure the Spark parameters for AWS under

Data access configuration

Add the following Spark configuration parameters and restart the SQL warehouse:

spark.hadoop.fs.s3a.access.key <S3 Access Key value>

spark.hadoop.fs.s3a.secret.key <S3 Secret Key value>

spark.hadoop.fs.s3a.endpoint <S3 Staging Bucket endpoint value>

For example, the S3 staging bucket warehouse value is

s3.ap-south-1.amazonaws.com

Ensure that the configured access key and secret key have access to the S3 buckets where you store the data for Databricks Delta tables.

Watch

Comments

0 COMMENTS

We’d like to hear from you! Log in to comment.