Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Big Data Management
  3. Mappings in the Hadoop Environment
  4. Mapping Sources in the Hadoop Environment
  5. Mapping Targets in the Hadoop Environment
  6. Mapping Transformations in the Hadoop Environment
  7. Processing Hierarchical Data on the Spark Engine
  8. Configuring Transformations to Process Hierarchical Data
  9. Processing Unstructured and Semi-structured Data with an Intelligent Structure Model
  10. Stateful Computing on the Spark Engine
  11. Monitoring Mappings in the Hadoop Environment
  12. Mappings in the Native Environment
  13. Profiles
  14. Native Environment Optimization
  15. Cluster Workflows
  16. Connections
  17. Data Type Reference
  18. Function Reference
  19. Parameter Reference

Precision Loss Due to Data Overflow

Precision Loss Due to Data Overflow

Precision loss can occur for decimal data types that run in high-precision mode on the Hive engine.
In high precision mode, transformations on the Hive engine support precision up to 38 digits. If the result of an expression has precision that is higher than 38, data overflow occurs. If data overflow occurs, excess digits are truncated in the result of the expression. To prevent data overflow, the Hive engine preserves the integral part of the result and adjusts the scale to maintain the precision at a maximum value of 38.
To maintain the precision at a maximum value of 38, the Hive engine subtracts digits of precision until the precision is 38. The following rules describe how the Hive engine adjusts the scale:
  • The Hive engine subtracts the same number of digits from the scale if the number of digits subtracted from the precision is less than or equal to the scale.
  • The Hive engine does not change the scale if the number of digits subtracted from the precision is greater than the scale.
When the Hive engine reduces the scale, the result is less accurate.
For example, the following expression multiplies two decimal data types dec(38,2) and dec(2,2):
dec(38,2) * dec(2,2)
The precision and scale of the result is calculated according to the following expression:
dec(38+2,2+2) = dec(40,4)
Since the precision of the result is greater than 38, the Hive engine adjusts the scale to return the precision at a maximum value of 38. The following expression shows how the Hive engine adjusts the precision and scale:
dec(40-2,4-2) = dec(38,2)


Updated October 23, 2019