Communities
A collaborative platform to connect and grow with like-minded Informaticans across the globe
Product Communities
Connect and collaborate with Informatica experts and champions
Discussions
Have a question? Start a Discussion and get immediate answers you are looking for
User Groups
Customer-organized groups that meet online and in-person. Join today to network, share ideas, and get tips on how to get the most out of Informatica
Get Started
Community Guidelines
Knowledge Center
Troubleshooting documents, product guides, how to videos, best practices, and more
Knowledge Base
One-stop self-service portal for solutions, FAQs, Whitepapers, How Tos, Videos, and more
Support TV
Video channel for step-by-step instructions to use our products, best practices, troubleshooting tips, and much more
Documentation
Information library of the latest product documents
Velocity (Best Practices)
Best practices and use cases from the Implementation team
Learn
Rich resources to help you leverage full capabilities of our products
Trainings
Role-based training programs for the best ROI
Certifications
Get certified on Informatica products. Free, Foundation, or Professional
Product Learning Paths
Free and unlimited modules based on your expertise level and journey
Resources
Library of content to help you leverage the best of Informatica products
Tech Tuesdays Webinars
Most popular webinars on product architecture, best practices, and more
Product Availability Matrix
Product Availability Matrix statements of Informatica products
SupportFlash
Monthly support newsletter
Support Documents
Informatica Support Guide and Statements, Quick Start Guides, and Cloud Product Description Schedule
Product Lifecycle
End of Life statements of Informatica products
Ideas
Events
Change Request Tracking
Marketplace
English
  • Data Engineering Streaming 10.5.1
  • All Products

Help for Users

User Guide

Use the Informatica® Data Engineering Streaming User Guide to learn about the configuration, guidelines, and usage of streaming mappings on the Spark engine in a non-native environment. The guide also includes information about the transformation processing differences and the sources and targets that you can use in streaming mappings.

Additional Content

Common Content for Data Engineering 10.5.1

Getting Started

Informatica 10.2.1 Big Data Primer

Basic information about Informatica 10.2.1 Big Data products: Big Data Management, Big Data Quality, Enterprise Data Lake, Big Data Streaming, and Enterprise Data Catalog.

Informatica 10.2.2 Big Data Primer

Click through this primer to get basic information about each Big Data product, along with the services, tools, documentation, and resources associated with the product.

Configure

Configuring Disaster Recovery for Streaming Sources in Big Data Streaming 10.2.2

You can use a streaming mapping to read from streaming sources, such as Apache Kafka. This article describes about configuring disaster recovery for the streaming sources to restore the processing after a mapping failure.

Configuring Intelligent Streaming 10.2 For Kafka on MapR

When you install Intelligent Streaming on MapR, you can run streaming mappings to read from any streaming source or write data to any streaming target without any additional configuration except for Kafka. This article provides the configuration steps that you need to perform to run streaming mappings to read from or write to Kafka on MapR.

Configuring Kerberos-enabled Kafka for a Big Data Streaming Mapping in 10.2.2

You can use Kerberos-enabled Kafka as a source or target in a streaming mapping. This article describes how to configure Kerberos-enabled Kafka for a streaming mapping in Big Data Streaming 10.2.2.

Creating, Deploying, and Updating an Application in the Developer Tool

Create and deploy an application that contains mappings, workflows, and other application objects to make the objects accessible to users that want to leverage the data outside of the Developer tool. You can deploy the application to a Data Integration Service to run the objects, or to an application archive file to save a copy of the …

Enable SSL Authentication in Apache Kafka using Portecle

Apache Kafka clients and brokers can communicate over SSL using a dedicated port. However, to use SSL authentication in Apache Kafka you must enable it. This article describes the steps to enable SSL authentication in Apache Kafka using Portecle.

How to Use Topic Patterns in Kafka Data Objects

You can read from Kafka sources in streaming mappings. When you configure a Kafka data object to read from a Kafka topic, you can specify the topic name or use a regular expression for the topic name pattern. This article describes the how to specify a topic pattern and the guidelines to use for the topic pattern while creating the data objects.

How to Use Topic Patterns in MapRStreams Data Objects

You can read from MapR Streams in streaming mappings. When you configure the MapRStreams data object to read from a stream, you can specify the stream name or use a regular expression for the stream name pattern. This article describes the how to specify a stream pattern and the guidelines to use for the stream pattern while creating …

Identifying Indirect and Remote Dependencies for an Application Patch in the Developer Tool

An applicaton patch can inherit direct, indirect, and remote dependencies. You can identify direct dependencies based on design-time objects, but you must use both the design-time and run-time objects to identify indirect and remote dependencies. This article will present scenarios to demonstrate how you can use the application object …

Use Case: Using Stateful Variables to Version Data in Informatica® Big Data Streaming

You can use stateful variables to perform stateful computing in a streaming mapping. This article presents a use case about using stateful variables to version data. You can navigate the use case to understand how stateful variables can achieve stateful computing and to think about how you can employ stateful variables based on your unique requirements.

Optimize and Tune

Ingest bulk data from multiple Kafka topics into Hive or HDFS using streaming mapping

You can use Data Engineering Streaming mappings to capture data from multiple data sources and ingest it into a target. This article describes how you can ingest CDC data from multiple Kafka topics into Data Engineering systems in a single mapping and publish the data to either HDFS or Hive partitioned tables.

Intelligent Streaming Performance Tuning Guidelines

You can tune Intelligent Streaming for better performance. This article provides recommendations that you can use to tune hardware, memory, Spark configuration, and mapping configuration.

Performance Tuning Guidelines for Lookup Transformations in Streaming Mappings

When you use Lookup transformations in Streaming mappings, multiple factors impact mapping performance. You can optimize performance when you configure Lookup transformation properties and execution parameters in the Streaming mapping. This article provides recommendations that you can use to optimize mapping performance.

Sizing Guidelines and Performance Tuning for Big Data Streaming 10.2.1

You can tune Big Data Streaming for better performance. This article provides recommendations that you can use to tune hardware, Spark configuration, mapping configuration, and the Kafka cluster.

Sizing Guidelines and Performance Tuning for Big Data Streaming 10.2.2

You can tune Big Data Streaming for better performance. This article provides recommendations that you can use to tune hardware, Spark configuration, mapping configuration, and the Kafka cluster.

Sizing Guidelines and Performance Tuning for Data Engineering Streaming 10.4.0

You can tune Data Engineering Streaming for better performance. This article provides recommendations that you can use to tune hardware, Spark configuration, mapping configuration, and the Kafka cluster.

Sizing Guidelines and Performance Tuning for Intelligent Streaming

You can tune Intelligent Streaming for better performance. This article provides recommendations that you can use to tune hardware, Spark configuration, mapping configuration, and the Kafka cluster.

Tuning the Hardware and Hadoop Cluster for Informatica Big Data Products

You can tune the hardware and the Hadoop cluster for better performance of Informatica big data products. This article provides tuning recommendations for Hadoop administrators and system administrators who set up the Hadoop cluster and hardware for Informatica big data products.

Administration

Supported Upgrade Paths to Big Data 10.2.1 Service Pack 1

Informatica 10.2.1 Service Pack 1 contains various improvements and enhancements to the Informatica domain. Informatica provides a list of supported upgrade paths for users who want to upgrade their product. This article describes the supported upgrade paths to upgrade to Informatica 10.2.1 Service Pack 1.

Security

Enabling SAML Authentication with Azure Active Directory for Web Applications

You can enable users to log into Informatica web applications using single sign-on. This article explains how to configure single sign-on in an Informatica 10.5 domain using Security Assertion Markup Language (SAML) v2.0 and the Azure Active Directory identity provider.

Enabling SAML authentication with Okta SSO for web applications

You can enable users to log into Informatica web applications using single sign-on. This article explains how to configure single sign-on in an Informatica domain using Security Assertion Markup Language (SAML) v2.0 and the Okta SSO identity provider.

Enabling SAML Authentication with Oracle Access Manager for Web Applications

You can enable users to log into Informatica web applications using single sign-on. This article explains how to configure single sign-on in an Informatica 10.5 domain using Security Assertion Markup Language (SAML) v2.0 and the Oracle Access Manager version 12.2.1 identity provider.

Install and Upgrade

10.5.1 Upgrade Paths

When you upgrade from a previous version, follow the supported upgrade paths to ensure a smooth and successful upgrade. This article includes upgrade paths for all products supported in the 10.5.1 Informatica installer.

Deploy Data Engineering Streaming on the AWS Cloud Marketplace (10.5.1)

You can deploy Data Engineering Streaming on the Amazon Web Services (AWS) Marketplace. This deployment reference includes step-by-step instructions for deploying Data Engineering Streaming on the Amazon Web Services (AWS) Marketplace. It also includes information on prerequisites and how to troubleshoot common issues.

Deploy Data Engineering Streaming on the AWS Cloud Marketplace (10.5)

You can deploy Data Engineering Streaming on the Amazon Web Services (AWS) Marketplace. This deployment reference includes step-by-step instructions for deploying Data Engineering Streaming on the Amazon Web Services (AWS) Marketplace. It also includes information on prerequisites, and how to troubleshoot common issues.

Deploy Data Engineering Streaming on the AWS U.S. Intelligence Community Marketplace (10.5.1.1)

You can deploy Data Engineering Streaming on the Amazon Web Services (AWS) U.S. Intelligence Community Marketplace. This deployment reference includes step-by-step instructions for deploying Data Engineering Streaming on the AWS U.S. Intelligence Community Marketplace. It also includes information on prerequisites and troubleshooting.

Deploying the Informatica® Data Engineering Streaming 10.4.0 AWS Marketplace Solution

This deployment reference provides step-by-step instructions for deploying Informatica® Data Engineering Streaming on the Amazon Web Services Cloud Marketplace. Automated reference deployments use AWS CloudFormation templates to launch, configure, and run the AWS compute, network, storage, and other services required to deploy a specific …

Deploying the Informatica® Data Engineering Streaming 10.4.0 Microsoft Azure Marketplace Solution

The automated marketplace solution uses Azure Resource Manager to launch, configure, and run the Azure virtual machine, virtual network, and other services required to deploy a specific workload on Azure. This deployment reference provides step-by-step instructions for deploying Informatica Data Engineering Streaming on the Microsoft …

Deploy the Informatica® 10.5.1.1 Data Engineering Streaming on the AWS Cloud Marketplace

You can deploy Data Engineering Streaming on the Amazon Web Services (AWS) Marketplace. This deployment reference includes step-by-step instructions for deploying Data Engineering Streaming on the Amazon Web Services (AWS) Marketplace. It also includes information on prerequisites and how to troubleshoot common issues.

Deploy the Informatica® 10.5.3 Data Engineering Streaming Solution on the Microsoft Azure Marketplace

The automated marketplace solution uses Azure Resource Manager to launch, configure, and run the Azure virtual machine, virtual network, and other services required to deploy a specific workload on Azure. This deployment reference provides step-by-step instructions for deploying Informatica Data Engineering Streaming on the Microsoft …

Deploy the Informatica® Data Engineering Streaming Solution on the AWS Cloud Marketplace (10.4.1)

You can deploy Data Engineering Streaming on the Amazon Web Services (AWS) Marketplace. This deployment reference includes step-by-step instructions for deploying Data Engineering Streaming on the Amazon Web Services (AWS) Marketplace. It also includes information on prerequisites, and how to troubleshoot common issues.

Deploy the Informatica® Data Engineering Streaming solution on the AWS Cloud Marketplace (10.5.2)

You can deploy Data Engineering Streaming on the Amazon Web Services (AWS) Marketplace. This deployment reference includes step-by-step instructions for deploying Data Engineering Streaming on the Amazon Web Services (AWS) Marketplace. It also includes information on prerequisites and how to troubleshoot common issues.

Deploy the Informatica® Data Engineering Streaming solution on the AWS Cloud Marketplace (10.5.3)

You can deploy Data Engineering Streaming on the Amazon Web Services (AWS) Marketplace. This deployment reference includes step-by-step instructions for deploying Data Engineering Streaming on the Amazon Web Services (AWS) Marketplace. It also includes information on prerequisites and how to troubleshoot common issues.

Deploy the Informatica® Data Engineering Streaming Solution on the Microsoft Azure Marketplace (10.4.1)

The automated marketplace solution uses Azure Resource Manager to launch, configure, and run the Azure virtual machine, virtual network, and other services required to deploy a specific workload on Azure. This deployment reference provides step-by-step instructions for deploying Informatica Data Engineering Streaming on the Microsoft …

Deploy the Informatica® Data Engineering Streaming Solution on the Microsoft Azure Marketplace (10.4.1)

The automated marketplace solution uses Azure Resource Manager to launch, configure, and run the Azure virtual machine, virtual network, and other services required to deploy a specific workload on Azure. This deployment reference provides step-by-step instructions for deploying Informatica Data Engineering Streaming on the Microsoft …

Deploy the Informatica® Data Engineering Streaming Solution on the Microsoft Azure Marketplace (10.5.1)

The automated marketplace solution uses Azure Resource Manager to launch, configure, and run the Azure virtual machine, virtual network, and other services required to deploy a specific workload on Azure. This deployment reference provides step-by-step instructions for deploying Informatica Data Engineering Streaming on the Microsoft …

Deploy the Informatica® Data Engineering Streaming Solution on the Microsoft Azure Marketplace (10.5)

The automated marketplace solution uses Azure Resource Manager to launch, configure, and run the Azure virtual machine, virtual network, and other services required to deploy a specific workload on Azure. This deployment reference provides step-by-step instructions for deploying Informatica Data Engineering Streaming on the Microsoft …

How to Migrate Mappings from the Hive Engine

Effective in version 10.2.2, Informatica dropped support for the Hive engine. You can run mappings on the Blaze and Spark engines in the Hadoop environment or on the Databricks Spark engine in the Databricks environment. This article tells how to change the validation and run-time environments for mappings, and it describes processing …

Additional Articles

FAQ: Authentication on HDInsight with Enterprise Security Package

FAQ: Authentication on HDInsight with Enterprise Security Package

Additional Content

Common Content for Data Engineering 10.5.1
Actions
Download Documentation Set
Send Feedback
Resources
Communities
Knowledge Base
Success Portal