Table of Contents

Search

  1. Preface
  2. Part 1: Version 10.4.1
  3. Part 2: Version 10.4.0
  4. Part 3: Version 10.2.2
  5. Part 4: Version 10.2.1
  6. Part 5: Version 10.2
  7. Part 6: Version 10.1.1
  8. Part 7: Version 10.1

Spark Engine

Spark Engine

Effective in version 10.1.1, the Spark engine has the following new features:

Binary Data Types

Effective in version 10.1.1, the Spark engine supports binary data type for the following functions:
  • DEC_BASE64
  • ENC_BASE64
  • MD5
  • UUID4
  • UUID_UNPARSE
  • CRC32
  • COMPRESS
  • DECOMPRESS (ignores precision)
  • AES Encrypt
  • AES Decrypt
The Spark engine does not support binary data type for the join and lookup conditions.
For more information, see the "Function Reference" chapter in the
Informatica Big Data Management 10.1.1 User Guide
.

Transformation Support on the Spark Engine

Effective in version 10.1.1, transformations have the following additional support on the Spark engine:
  • The Java transformation is supported with some restrictions.
  • The Lookup transformation can access a Hive lookup source.
For more information, see the "Mapping Objects in the Hadoop Environment" chapter in the
Informatica Big Data Management 10.1.1 User Guide
.

Run-time Statistics for Spark Engine Job Runs

Effective in version 10.1.1, you can view summary and detailed statistics for mapping jobs run on the Spark engine.
You can view the following Spark summary statistics in the
Summary Statistics
view:
  • Source. The name of the mapping source file.
  • Target. The name of the target file.
  • Rows. The number of rows read for source and target.
The
Detailed Statistics
view displays a graph of the row counts for Spark engine job runs.
For more information, see the "Mapping Objects in the Hadoop Environment" chapter in the
Informatica Big Data Management 10.1.1 User Guide
.

0 COMMENTS

We’d like to hear from you!