Communities
A collaborative platform to connect and grow with like-minded Informaticans across the globe
Product Communities
Connect and collaborate with Informatica experts and champions
Discussions
Have a question? Start a Discussion and get immediate answers you are looking for
User Groups
Customer-organized groups that meet online and in-person. Join today to network, share ideas, and get tips on how to get the most out of Informatica
Get Started
Community Guidelines
Knowledge Center
Troubleshooting documents, product guides, how to videos, best practices, and more
Knowledge Base
One-stop self-service portal for solutions, FAQs, Whitepapers, How Tos, Videos, and more
Support TV
Video channel for step-by-step instructions to use our products, best practices, troubleshooting tips, and much more
Documentation
Information library of the latest product documents
Velocity (Best Practices)
Best practices and use cases from the Implementation team
Learn
Rich resources to help you leverage full capabilities of our products
Trainings
Role-based training programs for the best ROI
Certifications
Get certified on Informatica products. Free, Foundation, or Professional
Product Learning Paths
Free and unlimited modules based on your expertise level and journey
Resources
Library of content to help you leverage the best of Informatica products
Tech Tuesdays Webinars
Most popular webinars on product architecture, best practices, and more
Product Availability Matrix
Product Availability Matrix statements of Informatica products
SupportFlash
Monthly support newsletter
Support Documents
Informatica Support Guide and Statements, Quick Start Guides, and Cloud Product Description Schedule
Product Lifecycle
End of Life statements of Informatica products
Ideas
Events
Change Request Tracking
Marketplace
English
  • Informatica Data Quality 10.5
  • All Products

Table of Contents

Search

  1. Preface
  2. Part 1: Version 10.5
  3. Part 2: Versions 10.4.1 - 10.4.1.3
  4. Part 3: Versions 10.4 - 10.4.0.2
  5. Part 4: Versions 10.2.2 - 10.2.2 HotFix 1
  6. Part 5: Version 10.2.1
  7. Part 6: Version 10.2 - 10.2 HotFix 2

What's New and Changed

What's New and Changed

Back Next

Transformation Support on the Spark Engine

Transformation Support on the Spark Engine

This section describes new transformation features on the Spark engine in version 10.2.1.

Transformation Support

Effective in version 10.2.1, the following transformations are supported on the Spark engine:
  • Case Converter
  • Classifier
  • Comparison
  • Key Generator
  • Labeler
  • Merge
  • Parser
  • Python
  • Standardizer
  • Weighted Average
Effective in version 10.2.1, the following transformations are supported with restrictions on the Spark engine:
  • Address Validator
  • Consolidation
  • Decision
  • Match
  • Sequence Generator
Effective in version 10.2.1, the following transformation has additional support on the Spark engine:
  • Java. Supports complex data types such as array, map, and struct to process hierarchical data.
For more information on transformation support, see the "Mapping Transformations in the Hadoop Environment" chapter in the
Informatica Big Data Management 10.2.1 User Guide
.
For more information about transformation operations, see the
Informatica 10.2.1 Developer Transformation Guide
.

Python Transformation

Effective in version 10.2.1, you can create a Python transformation in the Developer tool. Use the Python transformation to execute Python code in a mapping that runs on the Spark engine.
You can use a Python transformation to implement a machine model on the data that you pass through the transformation. For example, use the Python transformation to write Python code that loads a pre-trained model. You can use the pre-trained model to classify input data or create predictions.
The Python transformation is available for technical preview. Technical preview functionality is supported but is not production-ready. Informatica recommends that you use in non-production environments only.
For more information, see the "Python Transformation" chapter in the
Informatica 10.2.1 Developer Transformation Guide
.

Update Strategy Transformation

Effective in version 10.2.1, you can use Hive MERGE statements for mappings that run on the Spark engine to perform update strategy tasks. Using MERGE in queries is usually more efficient and helps increase performance.
Hive MERGE statements are supported for the following Hadoop distributions:
  • Amazon EMR 5.10
  • Azure HDInsight 3.6
  • Hortonworks HDP 2.6
To use Hive MERGE, select the option in the advanced properties of the Update Strategy transformation.
Previously, the Data Integration Service used INSERT, UPDATE and DELETE statements to perform this task using any run-time engine. The Update Strategy transformation still uses these statements in the following scenarios:
  • You do not select the Hive MERGE option.
  • Mappings run on the Hive or Blaze engine.
  • If the Hadoop distribution does not support Hive MERGE.
For more information about using a MERGE statement in Update Strategy transformations, see the chapter on Update Strategy transformation in the
Informatica Big Data Management 10.2.1 User Guide
.
0 COMMENTS
We’d like to hear from you!