Table of Contents

Search

  1. Preface
  2. Introduction to Intelligent Streaming
  3. Intelligent Streaming Configuration
  4. Connections
  5. Sources and Targets in a Streaming Mapping
  6. Intelligent Streaming Mappings
  7. Window Transformation
  8. Data Type Reference

Big Data Streaming User Guide

Big Data Streaming User Guide

Prerequisites

Prerequisites

Before you create an AmazonKinesis connection, perform the following tasks:
  1. Generate an Access Key ID and Secret Access Key for the user in AWS. You can provide these key values when you create an AmazonKinesis connection.
    If you do not provide these key values, the values are picked up from the AWS configuration file location on the Developer client machine, during metadata import. Similarly, at runtime, the values are picked up from the default AWS configuration file location on the Hadoop cluster.
  2. If you use the Amazon Kinesis Firehose service as a target, perform the following tasks:
    1. Verify that you have an AWS account with the required IAM permissions to use the AWS services Kinesis Firehose, S3, Redshift, and Elastic Search.
    2. If you are writing to Amazon Redshift, verify that you have the Redshift INSERT privilege to copy data from the Amazon S3 bucket to the Redshift cluster.
    3. Define a Firehose Delivery Stream with either S3 or Redshift or Elasticsearch as its destination.