Hi, I'm Ask INFA!
What would you like to know?
ASK INFAPreview
Please to access Ask INFA.

Table of Contents

Search

  1. Preface
  2. Introduction to Databricks Connector
  3. Connections for Databricks
  4. Mappings for Databricks
  5. Migrating a mapping
  6. SQL ELT with Databricks Connector
  7. Data type reference
  8. Troubleshooting

Databricks Connector

Databricks Connector

Databricks and transformation data types

Databricks and transformation data types

The following table compares the Databricks native data type to the transformation data type:
Databricks Data Type
Transformation Data Type
Range and Description
Binary
Binary
1 to 104,857,600 bytes.
Bigint
Bigint
-9,223,372,036,854,775,808 to +9,223,372,036,854,775,807.
8-byte signed integer.
Boolean
Integer
1 or 0.
Date
Date/Time
Date and time values.
Decimal
Decimal
Exact numeric of selectable precision
For mappings, the maximum precision is 28 and the scale is 27.
Double
Double
Precision 15.
Float
Double
Precision 7.
Int
Integer
-2,147,483,648 to +2,147,483,647.
Smallint
Integer
-32,768 to +32,767.
String
String
1 to 104,857,600 characters.
Tinyint
Integer
-128 to 127
Timestamp
Date/Time
January 1,0001 00:00:00 to December 31,9999 23:59:59.997443.
Timestamp values only preserve results up to microsecond precision of six digits. The precision beyond six digits is discarded.
Varchar
String
1 to 104,857,600 characters.
To use the precision of the Varchar data type set in Databricks and not the precision of the String transformation data type, set the
-Ddatabricks.honorNativePrecision=true
property for the Secure Agent.
For more information on how to configure the property, see the Varchar data type Knowledge Base article.

Rules and guidelines for data types

Consider the following rules and guidelines for data types:
  • You cannot read and write hierarchical data of decimal or varchar data type.
  • In mappings, if the decimal data type exceeds the precision of 28 in the source, the numeric value for the decimal is rounded off after the 18th precision and the remaining digits are replaced with zeroes in the target.
    For example, the value 1234567890123456789012345678.9012345678 from the source is rounded off to 1234567890123456900000000000 in the target.
    To resolve the issue in mappings, specify a precision that is less than or equal to 28 for the Decimal data type in the source table.

0 COMMENTS

We’d like to hear from you!