Table of Contents


  1. Preface
  2. Part 1: Hadoop Integration
  3. Part 2: Databricks Integration
  4. Appendix A: Connections Reference

Step 1. Create an IAM Role and Policy for S3 Access

Step 1. Create an IAM Role and Policy for S3 Access

Using S3 resources with mappings that run on a Databricks cluster requires you to create IAM roles and policies in your AWS account.
  1. Log in to the AWS account that has administrator access to the Databricks cluster that you want to integrate with Data Engineering Integration.
  2. Optionally create a new IAM role to correspond to an S3 access policy. You can create a policy or use an existing one.
    Use the following steps to create the IAM role:
    1. Select the IAM service, then
      Create Role
      The Create Roles wizard opens.
    2. Under
      Select type of trusted entity
      , select
      AWS service
    3. Under
      Choose the service that will use this role
      , select
    4. Click
      Next: Permissions
      Next: Tags
      Next: Review
    5. In the Role name field, type a role name, then click
      Create role
  3. Grant access to the S3 bucket that contains the resources for the mapping to access. To do this, attach an inline policy to the role that you want to use.
    1. In the
      tab, click
      Inline policy
    2. Select the
    3. Paste the following JSON statement:
      { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::<s3-bucket-name>" ] }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:PutObjectAcl" ], "Resource": [ "arn:aws:s3:::<s3-bucket-name>/*" ] } ] }
    4. Edit the pasted JSON statement with the name of the S3 bucket, then click
    5. In the
      field, type a name for the policy, then click
      Create policy
    window opens.
  4. In the
    window, select and copy the Instance Profile ARN string for use later in this process.
    The following image shows the
    window with redacted ARN strings:


We’d like to hear from you!