Table of Contents

Search

  1. Preface
  2. Introduction to Big Data Streaming
  3. Big Data Streaming Configuration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Appendix A: Connections
  9. Appendix B: Data Type Reference
  10. Appendix C: Sample Files

Big Data Streaming User Guide

Big Data Streaming User Guide

Prerequisites

Prerequisites

Before you create an AmazonKinesis connection, perform the following tasks:
  1. Generate an Access Key ID and Secret Access Key for the user in AWS. You can provide these key values when you create an AmazonKinesis connection.
    If you do not provide these key values, the values are picked up from the AWS configuration file location on the Developer client machine, during metadata import. Similarly, at runtime, the values are picked up from the default AWS configuration file location on the Hadoop cluster.
  2. If you use Amazon Kinesis Streams as a source, perform the following task:
    1. Verify that the credentials that belong IAM user that you specify in the Access Key ID has the consumer permissions that are part of the IAM policy.
      For the list of permissions, see the AWS documentation at https://docs.aws.amazon.com/streams/latest/dev/learning-kinesis-module-one-iam.html
    2. Verify that you have the following permissions to fetch metadata: kinesis:DescribeStream, kinesis:GetShardIterator, and kinesis:GetRecords
  3. If you use the Amazon Kinesis Firehose service as a target, perform the following tasks:
    1. Verify that you have an AWS account with the required IAM permissions to use the AWS services Kinesis Firehose, S3, Redshift, and Elastic Search.
    2. If you are writing to Amazon Redshift, verify that you have the Redshift INSERT privilege to copy data from the Amazon S3 bucket to the Redshift cluster.
    3. Define a Firehose Delivery Stream with either S3 or Redshift or Elasticsearch as its destination. Configure source as Direct PUT or other sources.
    4. Verify that the IAM user credentials have required permissions, based on the target you are writing to.