Table of Contents


  1. Preface
  2. Introduction to Data Engineering Streaming
  3. Data Engineering Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Transformation in Streaming Mappings
  8. Window Transformation
  9. Appendix A: Connections
  10. Appendix B: Monitoring REST API Reference
  11. Appendix C: Sample Files

Create AWS Credentials

Create AWS Credentials

Create AWS credentials for the users, such as access key ID and secret access key. An authentication type can be selected by the user during the creation of an Amazon Kinesis connection, such as AWS credential profile and cross-account IAM role. The default authentication type is AWS credential profile.

Access Key and ID

Generate an Access Key ID and Secret Access Key for the users in AWS.

AWS Credential Profile

You can define AWS credential profiles in the credentials file. Each credential profile consists of secret access key and access key ID.
Users can use the AWS credential profile names to use different AWS credentials at run time than the AWS credentials that they specify when they create an Amazon Kinesis connection with an Amazon Kinesis Streams as a source and Amazon Kinesis Firehose as a target.
Place the credentials file in the same location on all the cluster nodes. Default location is
<yarn home directory>/.aws/credentials
AWS credential profile is used at run time only.

Cross-account IAM Role

You can create an IAM role to allow a user of an AWS account to access resources of another AWS account. IAM roles allow you to define a set of permissions to access the AWS resources.
Users can share resources in one AWS account with users in a different AWS account without the need to create individual IAM users in each AWS account.
Cross-account IAM role is not supported for Amazon Kinesis Firehose as target.


We’d like to hear from you!