Consider the following rules and guidelines for data types:
Avro data types support:
Date, Decimal, and Timestamp data types are applicable when you run a mapping in the native environment or on the Spark engine in Cloudera CDH 6.1 distribution.
Time data type is applicable when you run a mapping in the native environment in Cloudera CDH 6.1 distribution.
JSON data types support:
For PowerExchange for Microsoft Azure Data Lake Storage Gen2, you can read and write complex file objects in JSON format in mappings that run in the native environment, Spark engine, and Databricks Spark engine.
For other file-based adapters, you can read and write complex file objects in JSON format in mappings that run on the Spark engine only.
Parquet data types support:
Date, Time, and Timestamp data types till microseconds are applicable when you run a mapping in the native environment , Blaze, and Spark engine in the Cloudera CDH 6.1, Hortonworks HDP 3.1, Azure HDInsight HDI 4.0, and Cloudera CDP 7.1 distributions.
Date,Time_Millis and Timestamp_Millis data types are applicable when you run a mapping in the native environment or Spark engine in the Amazon Elastic MapReduce (EMR) 5.23, MapR 6.1, and Hortonworks HDP 2.6 distributions.
Decimal data types are applicable when you run a mapping in the native environment and Spark engine in Cloudera CDH 6.1, Cloudera CDH 6.3, Hortonworks HDP 2.6, Hortonworks HDP 3.1, Amazon EMR 5.20, Amazon EMR 5.23, MapR 6.1, Google Dataproc 1.4, and Azure HDInsight HDI 4.0 distributions.
Date, Time, Timestamp, and Decimal data types are applicable when you run a mapping on the Databricks Spark engine.
When you run a mapping and use Date data type that does not have a time value, the Data Integration Service adds the time value, based on the time zone, to the date in the target.
For example, Date data type used in the source:
1980-01-09
Value generated in the target:
1980-01-09 00:00:00
When you run a mapping in the native environment and use Time data type in the source, the Data Integration Service writes incorrect date value to the target.
For example, Time data type used in the source:
1980-01-09 06:56:01.365235000
Incorrect Date value is generated in the target:
1899-12-31 06:56:01.365235000
When you run a mapping in the native environment and use Date data type in the source, the Data Integration Service writes incorrect time value to the target.
For example, Date data type used in the source:
1980-01-09 00:00:00
Incorrect Time value generated in the target:
1980-01-09 05:30:00
To run a mapping that reads and writes Date, Time, Timestamp, and Decimal data types, update the