Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Big Data Management
  3. Mappings in the Hadoop Environment
  4. Mapping Sources in the Hadoop Environment
  5. Mapping Targets in the Hadoop Environment
  6. Mapping Transformations in the Hadoop Environment
  7. Processing Hierarchical Data on the Spark Engine
  8. Configuring Transformations to Process Hierarchical Data
  9. Processing Unstructured and Semi-structured Data with an Intelligent Structure Model
  10. Stateful Computing on the Spark Engine
  11. Monitoring Mappings in the Hadoop Environment
  12. Mappings in the Native Environment
  13. Profiles
  14. Native Environment Optimization
  15. Cluster Workflows
  16. Connections
  17. Data Type Reference
  18. Function Reference
  19. Parameter Reference

Big Data Management User Guide

Big Data Management User Guide

Parquet and Transformation Data Types

Parquet and Transformation Data Types

Apache Parquet data types map to transformation data types that the Data Integration Service uses to move data across platforms.
The following table compares Parquet data types and transformation data types:
Parquet
Transformation
Description
Binary
Binary
1 to 104,857,600 bytes
Binary (UTF8)
String
1 to 104,857,600 characters
Boolean
Integer
-2,147,483,648 to 2,147,483,647
Precision of 10, scale of 0
Double
Double
Precision of 15 digits
Fixed Length Byte Array
Decimal
Decimal value with declared precision and scale. Scale must be less than or equal to precision.
For transformations that support precision up to 38 digits, the precision is 1 to 38 digits, and the scale is 0 to 38.
For transformations that support precision up to 28 digits, the precision is 1 to 28 digits, and the scale is 0 to 28.
If you specify the precision greater than the maximum number of digits, the Data Integration Service converts decimal values to double in high precision mode.
Float
Double
Precision of 15 digits
group (LIST)
Array
Unlimited number of characters.
Int32
Integer
-2,147,483,648 to 2,147,483,647
Precision of 10, scale of 0
Int64
Bigint
-9,223,372,036,854,775,808 to 9,223,372,036,854,775,807
Precision of 19, scale of 0
Int64 (TIMESTAMP_MILLIS)
Date/Time
Jan 1, 0001 A.D. to Dec 31, 9999 A.D.
Precision of 29, scale of 9
(precision to the nanosecond)
Combined date/time value.
Int96
Date/Time
Jan 1, 0001 A.D. to Dec 31, 9999 A.D.
Precision of 29, scale of 9
(precision to the nanosecond)
Combined date/time value.
Map
Map
Unlimited number of characters.
Struct
Struct
Unlimited number of characters.
Union
Corresponding primitive data type in a union of ["primitive_type", "null"] or ["null", "primitive_type"].
Dependent on primitive data type.

0 COMMENTS

We’d like to hear from you!