Table of Contents

Search

  1. Preface
  2. Introduction to Intelligent Streaming
  3. Intelligent Streaming Configuration
  4. Connections
  5. Sources and Targets in a Streaming Mapping
  6. Intelligent Streaming Mappings
  7. Window Transformation
  8. Data Type Reference

AmazonKinesis Data Objects

AmazonKinesis Data Objects

An AmazonKinesis data object is a physical data object that represents data in a Amazon Kinesis Stream or Amazon Kinesis Firehose Delivery Stream. Create an AmazonKinesis data object to read from Amazon Kinesis Streams or to write to Amazon Kinesis Firehose.
After you create an AmazonKinesis connection, create an AmazonKinesis data object.
When you configure the AmazonKinesis data object, specify the name of the Amazon Kinesis Stream that you read from or the Firehose Delivery Stream that you write to. You can specify the Kinesis Firehose Delivery Stream name or use a regular expression for the stream name pattern. If the input has multiple partitions, you can create multiple Kinesis Firehose Delivery Streams to the same target and send the data from these partitions to the individual delivery streams based on the pattern you specify in the stream name.
After you create the data object, create a read operation to read data from an Amazon Kinesis Stream or a data object write operation to write data to an Amazon Kinesis Firehose Delivery Stream. You can then add the data object read operation as a source and the data object write operation as a target in Streaming mappings.
When you configure the data operation properties, specify the format in which the data object reads or writes data. When you read from Amazon Kinesis Stream sources, you can read data in JSON, XML, Avro, Flat, or binary format. When you write to Amazon Firehose targets, you can specify JSON or binary as the format.
When you specify XML format, you must provide a XSD file. When you specify JSON, Avro, or Flat format, you must provide a sample file.
You can pass any payload format directly from source to target in Streaming mappings. You can project columns in binary format pass a payload from source to target in its original form or to pass a payload format that is not supported.
Streaming mappings can read, process, and write hierarchical data. You can use array, struct, and map complex data types to process the hierarchical data. You assign complex data types to ports in a mapping to flow hierarchical data. Ports that flow hierarchical data are called complex ports.
When you run a mapping to write data to an Amazon Kinesis Firehose Delivery Stream, the data object uses the AWS Firehose SDK to write data.
You cannot run a mapping with an AmazoznKinesis data object on a MapR distribution.
For more information about processing hierarchical data, see the
Informatica Big Data Management User Guide
.