Communities
A collaborative platform to connect and grow with like-minded Informaticans across the globe
Product Communities
Connect and collaborate with Informatica experts and champions
Discussions
Have a question? Start a Discussion and get immediate answers you are looking for
User Groups
Customer-organized groups that meet online and in-person. Join today to network, share ideas, and get tips on how to get the most out of Informatica
Get Started
Community Guidelines
Knowledge Center
Troubleshooting documents, product guides, how to videos, best practices, and more
Knowledge Base
One-stop self-service portal for solutions, FAQs, Whitepapers, How Tos, Videos, and more
Support TV
Video channel for step-by-step instructions to use our products, best practices, troubleshooting tips, and much more
Documentation
Information library of the latest product documents
Velocity (Best Practices)
Best practices and use cases from the Implementation team
Learn
Rich resources to help you leverage full capabilities of our products
Trainings
Role-based training programs for the best ROI
Certifications
Get certified on Informatica products. Free, Foundation, or Professional
Product Learning Paths
Free and unlimited modules based on your expertise level and journey
Resources
Library of content to help you leverage the best of Informatica products
Tech Tuesdays Webinars
Most popular webinars on product architecture, best practices, and more
Product Availability Matrix
Product Availability Matrix statements of Informatica products
SupportFlash
Monthly support newsletter
Support Documents
Informatica Support Guide and Statements, Quick Start Guides, and Cloud Product Description Schedule
Product Lifecycle
End of Life statements of Informatica products
Ideas
Events
Change Request Tracking
Marketplace
English
  • Data Engineering Integration 10.5.4.1
  • All Products

Table of Contents

Search

  1. Preface
  2. Part 1: Hadoop Integration
  3. Part 2: Databricks Integration
  4. Appendix A: Managing Distribution Packages
  5. Appendix B: Connections Reference

Integration Guide

Integration Guide

Back Next

Configure S3 and Redshift Authentication and Encryption on AWS

Configure S3 and Redshift Authentication and Encryption on AWS

If the Databricks cluster uses the AWS S3 service to access sources or targets stored in S3 buckets or the Redshift data warehouse, configuration depends on your choices for authentication and encryption.

Authentication Options

Configure the following authentication options:
Key Pair Authentication
If the data resources use AWS key pair authentication, add the access key and the secret key in the Spark tab of the Advanced Configuration section of the Databricks cluster configuration page. Separate each key value with a space.
The image below shows an example of the keys in the text entry pane of the Spark tab:
The image shows a text entry pane under the Spark tab. Under "Spark Config," two key pairs have been entered.
IAM Role Authentication
If the data resources use IAM role authentication, verify that the configuration meets one of the following requirements:
  • The S3 bucket belongs to the AWS account in which the Databricks cluster resides.
  • The S3 bucket belongs to a different AWS account than the one in which the Databricks cluster resides, and you enabled a cross-account policy to allow the cluster to access the bucket.
For more information about using IAM roles with a Databricks cluster, see the AWS documentation.

Encryption Options

Choose from the following types of encryption. Each can be combined with either of the two options for authentication.
Configure the following properties in the Spark configuration tab of the cluster:
Server-Side S3 Encryption (SSE-S3)
Configure the following property to enable SSE-S3 encryption: 
spark.hadoop.fs.s3a.server-side-encryption-algorithm AES256
If you use both key pair authentication and SSE-S3 encryption, then add this property in the Spark configuration tab after the first two lines for key pair authentication. For example: 
spark.hadoop.fs.s3n.awsAccessKeyld YYYYXXXX 
spark.hadoop.fs.s3n.awsSecretAccessKey XYYYYSSSSSSS
spark.hadoop.fs.s3a.server-side-encryption-algorithm AES256
Server-Side Encryption with KMS (SSE-KMS)
Configure the following properties to enable SSE-KMS encryption: 
spark.hadoop.fs.s3a.server-side-encryption-kms-master-key-id arn:aws:kms:us-west-XXXXX:key/XXXYYYYYYYYYYYYYYYYY 
spark.hadoop.fs.s3a.server-side-encryption-algorithm aws:kms 
spark.hadoop.fs.s3a.impl com.data bricks.s3a.S3AFileSystem
If you use both key pair authentication and SSE-KMS encryption, then add these properties in the Spark configuration tab after the first two lines for key pair authentication. For example: 
spark.hadoop.fs.s3n.awsAccessKeyld YYYYXXXX  
spark.hadoop.fs.s3n.awsSecretAccessKey XYYYYSSSSSSS
spark.hadoop.fs.s3a.server-side-encryption-kms-master-key-id arn:aws:kms:us-west-XXXXX:key/XXXYYYYYYYYYYYYYYYYY 
spark.hadoop.fs.s3a.server-side-encryption-algorithm aws:kms 
spark.hadoop.fs.s3a.impl com.data bricks.s3a.S3AFileSystem
0 COMMENTS
We’d like to hear from you!