Communities
A collaborative platform to connect and grow with like-minded Informaticans across the globe
Product Communities
Connect and collaborate with Informatica experts and champions
Discussions
Have a question? Start a Discussion and get immediate answers you are looking for
User Groups
Customer-organized groups that meet online and in-person. Join today to network, share ideas, and get tips on how to get the most out of Informatica
Get Started
Community Guidelines
Knowledge Center
Troubleshooting documents, product guides, how to videos, best practices, and more
Knowledge Base
One-stop self-service portal for solutions, FAQs, Whitepapers, How Tos, Videos, and more
Support TV
Video channel for step-by-step instructions to use our products, best practices, troubleshooting tips, and much more
Documentation
Information library of the latest product documents
Velocity (Best Practices)
Best practices and use cases from the Implementation team
Learn
Rich resources to help you leverage full capabilities of our products
Trainings
Role-based training programs for the best ROI
Certifications
Get certified on Informatica products. Free, Foundation, or Professional
Product Learning Paths
Free and unlimited modules based on your expertise level and journey
Resources
Library of content to help you leverage the best of Informatica products
Tech Tuesdays Webinars
Most popular webinars on product architecture, best practices, and more
Product Availability Matrix
Product Availability Matrix statements of Informatica products
SupportFlash
Monthly support newsletter
Support Documents
Informatica Support Guide and Statements, Quick Start Guides, and Cloud Product Description Schedule
Product Lifecycle
End of Life statements of Informatica products
Ideas
Events
Change Request Tracking
Marketplace
English
  • Data Engineering Integration 10.4.1
  • All Products

Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Data Engineering Integration
  3. Mappings
  4. Mapping Optimization
  5. Sources
  6. Targets
  7. Transformations
  8. Python Transformation
  9. Data Preview
  10. Cluster Workflows
  11. Profiles
  12. Monitoring
  13. Hierarchical Data Processing
  14. Hierarchical Data Processing Configuration
  15. Hierarchical Data Processing with Schema Changes
  16. Intelligent Structure Models
  17. Blockchain
  18. Stateful Computing
  19. Appendix A: Connections Reference
  20. Appendix B: Data Type Reference
  21. Appendix C: Function Reference

User Guide

User Guide

Back Next

Function and Data Type Processing on the Spark Engine

Function and Data Type Processing on the Spark Engine

Some restrictions and guidelines apply to processing Informatica functions on the Spark engine.
Consider the following rules and guidelines for function and data type processing on the Spark engine:
  • The Spark engine and the Data Integration Service process overflow values differently. The Spark engine processing rules might differ from the rules that the Data Integration Service uses. As a result, mapping results can vary between the native and Hadoop environment when the Spark engine processes an overflow.
    Consider the following processing variation for Spark:
    • If an expression results in numerical errors, such as division by zero or SQRT of a negative number, the Spark engine returns NULL. In the native environment, the same expression results in a row error.
  • The Spark engine and the Data Integration Service process data type conversions differently. As a result, mapping results can vary between the native and Hadoop environment when the Spark engine performs a data type conversion. Consider the following processing variations for Spark:
    • The results of arithmetic operations on floating point types, such as Decimal, can vary between the native environment and a Hadoop environment. The difference between the results can increase across multiple operations.
    • When you use the TO_DECIMAL or TO_DECIMAL38 function in a mapping that runs in high-precision mode, you must specify a scale argument. If the mapping runs in low-precision mode, the Spark engine ignores the scale argument and returns a double.
    • When the number of fractional digits in a double or decimal value exceeds the scale that is configured in a decimal port, the Spark engine trims trailing digits, rounding the value if necessary.
    • If you use Hive 2.0 or higher, the Spark engine guarantees scale values. For example, when the Spark engine processes the decimal
      1.1234567
       with scale 9 using Hive 2.0, the output is
      1.123456700
      .
    • The Spark engine cannot process dates to the nanosecond. It can return a precision for date/time data up to the microsecond.
  • The Spark engine and the Data Integration Service process scale differently. The Data Integration Service allows scale to differ between rows of decimal data while the Spark engine uses a fixed scale for each row. Because the scale is fixed, arithmetic operations can result in data overflow.
    For example, the arithmetic operation dec(38,0) / dec(10,0) outputs a decimal dec(38,6) on the Spark engine. The operation might result in data overflow based on whether the result can be represented as a decimal dec(38,6).
    The following table shows the decimal values of dec(10,0) that result in data overflow for several decimal values of dec(38,0):
    Decimal Value of dec(38,0)
    Decimal Values of dec(10,0) that Result in Data Overflow
    Less than 10^32
    None
    10^32
    Decimals with an absolute value that is less than or equal to 1
    10^33
    Decimals with an absolute value that is less than or equal to 10
    10^34
    Decimals with an absolute value that is less than or equal to 100
  • If data overflow occurs, the Spark engine returns NULL. For example, if you use the expression
    TO_DECIMAL(12.34, 2)
     in a port that has a precision of 3 and a scale of 2, the return value is NULL. The NULL value will be propagated through the mapping. The mapping might overwrite it using a default value, detect it using the function IS_NULL, and write the NULL value to the target.
  • If you enable high-precision in a streaming mapping, the Spark engine runs the mapping in low-precision mode.
  • If the mapping contains a complex port with an element of a decimal data type, the Spark engine runs the mapping in low-precision mode.
  • The Hadoop environment treats "/n" values as null values. If an aggregate function contains empty or NULL values, the Hadoop environment includes these values while performing an aggregate calculation.
  • Mapping validation fails if you configure SYSTIMESTAMP with a variable value, such as a port name. The function can either include no argument or the precision to which you want to retrieve the timestamp value.
  • Mapping validation fails if an output port contains a Timestamp with Time Zone data type.
  • Avoid including single and nested functions in an Aggregator transformation. The Data Integration Service fails the mapping in the native environment. It can push the processing to the Hadoop environment, but you might get unexpected results. Informatica recommends creating multiple transformations to perform the aggregation.
  • You cannot preview data for a transformation that is configured for windowing.
  • The Spark METAPHONE function uses phonetic encoders from the
    org.apache.commons.codec.language
     library. When the Spark engine runs a mapping, the METAPHONE function can produce an output that is different from the output in the native environment. The following table shows some examples:
    String
    Data Integration Service
    Spark Engine
    Might
    MFT
    MT
    High
    HF
    H
  • If you use the TO_DATE function on the Spark engine to process a string written in ISO standard format, you must add
    *T*
     to the date string and
    *”T”*
     to the format string. The following expression shows an example that uses the TO_DATE function to convert a string written in the ISO standard format YYYY-MM-DDTHH24:MI:SS: 
    TO_DATE(‘2017-11-03*T*12:45:00’,’YYYY-MM-DD*”T”*HH24:MI:SS’)
    The following table shows how the function converts the string: 
    ISO Standard Format
    RETURN VALUE
    		 
    2017-11-03T12:45:00
    Nov 03 2017 12:45:00
  • The UUID4 function is supported only when used as an argument in UUID_UNPARSE or ENC_BASE64.
  • The UUID_UNPARSE function is supported only when the argument is UUID4( ).
0 COMMENTS
We’d like to hear from you!