Network
Data Engineering
Data Engineering Integration
Enterprise Data Catalog
Enterprise Data Preparation
Cloud Integration
Cloud Application Integration
Cloud Data Integration
Cloud Customer 360
DiscoveryIQ
Cloud Data Wizard
Informatica for AWS
Informatica for Microsoft
Cloud Integration Hub
Complex Event Processing
Proactive Healthcare Decision Management
Proactive Monitoring
Real-Time Alert Manager
Rule Point
Data Integration
B2B Data Exchange
B2B Data Transformation
Data Integration Hub
Data Replication
Data Services
Data Validation Option
Fast Clone
Informatica Platform
Metadata Manager
PowerCenter
PowerCenter Express
PowerExchange
PowerExchange Adapters
Data Quality
Axon Data Governance
Data as a Service
Data Explorer
Data Quality
Data Security Group (Formerly ILM)
Data Archive
Data Centric Security
Secure@Source
Secure Testing
Master Data Management
Identity Resolution
MDM - Relate 360
Multidomain MDM
MDM Registry Edition
Process Automation
ActiveVOS
Process Automation
Product Information Management
Informatica Procurement
MDM - Product 360
Ultra Messaging
Ultra Messaging Options
Ultra Messaging Persistence Edition
Ultra Messaging Queuing Edition
Ultra Messaging Streaming Edition
Edge Data Streaming
Knowledge Base
Resources
PAM (Product Availability Matrices)
Support TV
Velocity (Best Practices)
Mapping Templates
Debugging Tools
User Groups
Documentation
English
English
English
Español
Spanish
Deutsch
German
Français
French
日本語
Japanese
한국어
Korean
Português
Portuguese
中文
Chinese
Log Out
Data Engineering Streaming
10.5.2
10.5.2
10.5.1
10.5
10.4.1
10.4.0
10.2.2 Service Pack 1
10.2.2
10.2.1
10.2 HotFix 2
10.2 HotFix 1
10.2
H2L
User Guide
Data Engineering Streaming 10.5.2
Data Engineering Streaming 10.5.2
All Products
Rename Saved Search
Name
* This field is required
Overwrite saved search
Confirm Deletion
Are you sure you want to delete the saved search?
Table of Contents
Search
No Results
Preface
Introduction to Data Engineering Streaming
Data Engineering Streaming Overview
Streaming Process
Component Architecture
Clients and Tools
Application Services
Repository
Third-Party Applications
Data Engineering Streaming Engines
Example
Data Engineering Streaming Administration
Data Engineering Streaming Configuration Overview
Authorization Systems
Authentication Systems on Hadoop
Authentication Systems on Databricks
Configuration for Kerberized Kafka Clusters
Configure the Default Realm
Configure Java Authorization and Authentication Service (JAAS)
Configuration for Amazon Kinesis
Create AWS Credentials
Configuration for Amazon Kinesis Streams Sources
Configuration for Amazon Kinesis Firehose Targets
Configuration for Azure Event Hubs
Configuration for Snowflake Targets
Sources in a Streaming Mapping
Sources in a Streaming Mapping Overview
Sources in a Streaming Mapping on Hadoop and Databricks
Processing Hierarchical Data in Streaming Mappings
Amazon Kinesis Data Objects
Amazon Kinesis Data Object Overview Properties
Amazon Kinesis Header Ports
Amazon Kinesis Data Object Read Operation Properties
General Properties
Ports Properties
Sources Properties
Run-time Properties
Advanced Properties
Column Projections Properties
Configuring Schema for Flat Files
Azure Event Hubs Data Objects
Azure Event Hubs Data Object Overview Properties
Azure Event Hubs Header Ports
Azure Event Hubs Data Object Read Operation Properties
General Properties
Ports Properties
Sources Properties
Run-time Properties
Advanced Properties
Column Projections Properties
Confluent Kafka Data Objects
Confluent Kafka Data Object Overview Properties
Confluent Kafka Header Ports
Confluent Kafka Data Object Read Operation Properties
General Properties
Ports Properties
Sources Properties
Schema Properties
Run-time Properties
Advanced Properties
Google PubSub Data Objects
Google PubSub Data Object Overview Properties
Google PubSub Data Object Read Properties
General Properties
Ports Properties
Schema Properties
Run-time Properties
Advanced Properties
HBase Data Objects
HBase Data Object Read Operation Properties
JMS Data Objects
Integration with JMS
JMS Message Structure
Header Fields
Property Fields
Body Fields
JMS Data Object Overview Properties
JMS Data Object Read Operation Properties
General Properties
Ports Properties
Sources Properties
Run-time Properties
Advanced Properties
Column Projections Properties
Configuring a Schema for Flat Files
Kafka Data Objects
Kafka Data Object Overview Properties
Kafka Header Ports
Kafka Data Object Read Operation Properties
General Properties
Ports Properties
Run-time Properties
Advanced Properties
Sources Properties
Column Projection Properties
Configuring Schema for Flat Files
MapR Streams Data Objects
MapR Streams Object Overview Properties
MapR Streams Header Ports for a Source
MapR Streams Data Object Read Operation Properties
General Properties
Ports Properties
Sources Properties
Run-time Properties
Advanced Properties
Column Projections Properties
Targets in a Streaming Mapping
Targets in a Streaming Mapping Overview
Targets in a Streaming Mapping on Hadoop and Databricks
Processing Hierarchical Data in Streaming Mappings
Amazon Kinesis Data Objects
Amazon Kinesis Data Object Overview Properties
Amazon Kinesis Data Object Write Operation Properties
General Properties
Ports Properties
Target Properties
Run-time Properties
Advanced Properties
Column Projection Properties
Amazon S3 Data Objects
Amazon S3 Data Object Properties
Amazon S3 Data Object Write Operation Properties
General Properties
Ports Properties
Target Properties
Run-time Properties
Advanced Properties
Column Projection Properties
FileName Port in Amazon S3
Azure Event Hubs Data Objects
Azure Event Hubs Data Object Overview Properties
Azure Event Hubs Header Ports
Azure Event Hubs Data Object Write Operations
General Properties
Ports Properties
Target Properties
Run-time Properties
Advanced Properties
Column Projections Properties
Cassandra Data Objects
Cassandra Data Object Overview Properties
Cassandra Data Object Write Operation Properties
General Properties
Ports Properties
Run-time Properties
Advanced Properties
Cassandra, ODBC, and Transformation Data types
Complex File Data Objects
Complex File Data Object Overview Properties
Complex File Header Port
Compression and Decompression for Complex File Targets
Complex File Data Object Write Operation Properties
General Properties
Ports Properties
Target Properties
Run-time Properties
Advanced Properties
Column Projection Properties
Complex File Execution Parameters
Temporary Directory for the Complex File Target
FileName Port in Complex File
Confluent Kafka Data Objects
Confluent Kafka Data Object Overview Properties
Confluent Kafka Header Ports
Confluent Kafka Data Object Write Properties
General Properties
Ports Properties
Targets Properties
Schema Properties
Run-time Properties
Advanced Properties
Google Cloud Storage Data Objects
Google Cloud Storage Data Object Overview Properties
Google Cloud Storage Data Object Write Operation Properties
General Properties
Ports Properties
Schema Properties
Run-time Properties
Advanced Properties
HBase Data Objects
Data Object Column Configuration
Add Columns
Search and Add Columns
Get All Columns
HBase Object Overview Properties
HBase Data Object Write Operation Properties
Advanced Properties
JMS Data Objects
Integration with JMS
JMS Message Structure
Header Fields
Property Fields
Body Fields
JMS Data Object Overview Properties
JMS Data Object Write Operation Properties
General Properties
Ports Properties
Target Properties
Run-time Properties
Advanced Properties
Column Projections Properties
Kafka Data Objects
Kafka Data Object Overview Properties
Kafka Header Ports
Kafka Data Object Write Operation Properties
General Properties
Ports Properties
Run-time Properties
Target Properties
Advanced Properties
Column Projections Properties
Kudu Data Objects
Kudu Data Object Overview Properties
Kudu Data Object Write Operation Properties
General Properties
Ports Properties
Run-time Properties
Advanced Properties
MapR Streams Data Objects
MapR Streams Object Overview Properties
MapR Streams Header Ports for a Target
MapR Streams Data Object Write Operation Properties
General Properties
Port Properties
Target Properties
Run-time Properties
Column Projection Properties
Microsoft Azure Data Lake Storage Gen1 Data Object
Microsoft Azure Data Lake Storage Gen1 Data Object Properties
Microsoft Azure Data Lake Storage Gen1 Data Object Write Operation Properties
Input Properties of the Data Object Write Operation
Ports Properties - Input Write
Run-time Properties
Advanced Properties
Column Projection Properties
Microsoft Azure Data Lake Storage Gen2 Data Object
Microsoft Azure Data Lake Storage Gen2 Data Object Properties
Microsoft Azure Data Lake Store Gen2 Data Object Write Operation Properties
General Properties
Ports Properties
Targets Properties
Schema Properties
Run-time Properties
Advanced Properties
FileName Port in ADLS Gen2
Relational Data Objects
Relational Data Object Overview Properties
Relational Data Object Write Operation Properties
General Properties
Data Object Properties
Ports Properties
Run-time Properties
Advanced Properties
Snowflake Data Objects
Snowflake Data Object Write Operation Properties
General Properties
Port Properties
Run-time Properties
Advanced Properties
Streaming Mappings
Streaming Mappings Overview
Data Objects
Creating a Data Object
Creating a Data Object Operation
Stateful Computing
Partitioning Configuration
Example
Rules in a Streaming Mapping
Mapping Configurations for Databricks
Mapping Configurations for Hadoop
Dynamic Mappings
Confluent Kafka Dynamic Mapping
Refresh Schema
Rules and Guidelines for Dynamic Mapping
Mapping Validation
Environment Validation
Data Object Validation
Run-time Validation
Transformation Validation
Mapping Execution Plan
Monitor Jobs
Streaming Mapping Example
Cluster Workflows
Ephemeral Cluster in Streaming Mappings
High Availability Configuration
Troubleshooting Streaming Mappings
Transformation in Streaming Mappings
Transformations in a Streaming Mapping Overview
Address Validator Transformation in a Streaming Mapping
Aggregator Transformation in a Streaming Mapping
Classifier Transformation in a Streaming Mapping
Data Masking Transformation in a Streaming Mapping
Expression Transformation in a Streaming Mapping
Filter Transformation in a Streaming Mapping
Java Transformation in a Streaming Mapping
Joiner Transformation in a Streaming Mapping
Lookup Transformation in a Streaming Mapping
Macro Transformation in a Streaming Mapping
Normalizer Transformation in a Streaming Mapping
Parser Transformation in a Streaming Mapping
Python Transformation in a Streaming Mapping
Rank Transformation in a Streaming Mapping
Router Transformation in a Streaming Mapping
Sorter Transformation in a Streaming Mapping
Standardizer Transformation in a Streaming Mapping
Union Transformation in a Streaming Mapping
Window Transformation in a Streaming Mapping
Data Preview
Window Transformation
Window Transformation Overview
Window Transformation Types
Tumbling Window
Sliding Window
Window Transformation Port Properties
Window Transformation Window Properties
Tumbling Window Transformation Example
Sliding Window Transformation Example
Rules and Guidelines for the Window Transformations
Appendix A: Connections
Connections Overview
Amazon Kinesis Connection
General Properties
Connection Properties
Creating an Amazon Kinesis Connection Using infacmd
Amazon S3 Connection
General Properties
Connection Properties
Creating an Amazon S3 Connection Using infacmd
Azure Event Hub Connection
General Properties
Connection Properties
Creating an Azure Event Hub Connection Using infacmd
Cassandra Connection Properties
Confluent Kafka Connection
General Properties
Confluent Kafka Broker Properties
SSL Properties
Creating a Confluent Kafka Connection Using infacmd
Google Cloud Storage Connection Properties
Google PubSub Connection Properties
HBase Connection
General Properties
Connection Properties
Creating an HBASE Connection Using infacmd
HDFS Connection
General Properties
Connection Properties
Creating an HDFS Connection Using infacmd
Hive Connection
Creating a Hive Connection Using infacmd
JMS Connection
Prerequisites to Create a JMS Connection and a JMS Data Object
Prerequisites to Use a JMS Connection and a JMS Data Object
General Properties
Connection Properties
Creating a JMS Connection Using infacmd
JDBC Connection Properties
JDBC V2 Connection Properties
Kafka Connection Properties
General Properties
Kafka Broker Properties
SSL Properties
Creating a Kafka Connection Using infacmd
Kudu Connection Properties
Creating a Kudu Connection using Informatica Command Line Program
MapR Streams Connection
General Properties
Connection Properties
Creating a MapR Streams Connection Using infacmd
Microsoft Azure Data Lake Storage Gen1 Connection
Microsoft Azure Data Lake Storage Gen1 Connection Properties
Microsoft Azure Data Lake Storage Gen2 Connection
General Properties
Connection Properties
Creating a Microsoft Azure Data Lake Storage Gen2 Connection Using infacmd
Snowflake Connection
General Properties
Snowflake Connection Properties
Creating a Snowflake Connection using Informatica Command Line Program
Appendix B: Monitoring REST API Reference
Monitoring REST API Reference
Appendix C: Sample Files
Sample Files
Sample XSD File
Sample JSON Schema
Sample Avro Schema
Sample Flat Format Schema
Sample Parquet Schema
User Guide
User Guide
10.5.2
10.5.1
10.5
10.4.1
10.4.0
10.2.2 Service Pack 1
10.2.2
10.2.1
10.2 HotFix 2
10.2 HotFix 1
10.2
Back
Next
Rank Transformation in a Streaming Mapping
Rank Transformation in a Streaming Mapping
Streaming mappings have additional processing rules that do not apply to batch mappings.
Mapping Validation
Mapping validation fails in the following situations:
A Rank transformation is in the same streaming pipeline as a passive Lookup transformation configured with an inequality lookup condition.
A Rank transformation is upstream from a Joiner transformation.
A streaming pipeline contains more than one Rank transformation.
A streaming pipeline contains an Aggregator transformation and a Rank transformation.
Transformations in a Streaming Mapping Overview
Updated April 21, 2022
Download Guide
Send Feedback
Resources
Communities
Knowledge Base
Success Portal
Back to Top
Back
Next
