Metadata Manager 10.4.0
- Metadata Manager 10.4.0
- All Products
When you run a mapping in the native environment to write data to a complex file object in Avro format with fixed data type, the mapping runs successfully but the Data Integration Service writes incorrect data to the target.
Change the data type in target schema to bytes and run the mapping again.
A Sqoop mapping configured for incremental data extraction fails when the incremental key value for the -infa-incremental-key parameter is in lower case.
When you run a mapping in the native environment to write data to a complex file object in Parquet format with fixed_len_byte_array data type, the mapping runs successfully but the Data Integration Service writes incorrect data to the target.
Change the data type in target schema to binary and run the mapping again.
physicalType: int32 for precision <=9, int64 for 9< precision <= 18, as BINARY for precision>18.logicalType: decimal(precision, scale)
Intelligent Structure Discovery does not process the Union data type in ORC input. When you try to base a model on a sample ORC file that contains Union data, the model creation fails with the following error: "Failed to create the model. Select a different sample file."
When you use a Create Cluster task to create a Databricks cluster and you specify cluster tags in a file, the task fails with the following error:
In the cluster tags file contents, include a comma after the last key-value pair. For example:
If a mapping configured for HDP 3.1 or Dataproc 1.4 on the Spark engine reads data from a Hive table and you specify database names in both the Data Access Connection String and in the runtime properties, a SQL override uses the database specified in the Hive connection instead of the database in the data object.
When you specify a file path as the value for the Databricks Create Cluster task advanced properties Cluster Tags, Spark Configurations, or Environment Variables, the Create Cluster task fails with the following error:
If you select the target schema strategy
FAIL - Fail mapping if target schema is differentand truncate the target Hive table but the target schema from the mapping flow is the same as the schema of the target table, the Spark engine appends data to the target instead of truncating the target table.
When you cancel a data preview job on the Spark engine, the behavior is unpredictable.
Mappings configured to run using Blaze LLAP fail with "Table not found" error because the Blaze engine does not support LLAP.
Perform one of the following tasks to avoid the error:
When you enable the Data Integration Service for data engineering recovery, the Data Integration Service logs point to
BigDataJobRecoveryData Integration Service option instead of the
DataEngineeringRecoveryData Integration Service option as displayed on the Administrator tool.
Creating an SQL data service for a flat file data source in the Developer tool intermittently fails with null pointer exception.
When you preview data with an Expression transformation configured for windowing as the preview point, the job might fail.
Mapping validation fails on the Spark engine when you create a complex target from a transformation that includes a complex port and:
The Data Integration Service does not recover mappings configured to run on the Spark engine that have mapping outputs even if you enable data engineering recovery for the mapping.
When you enable data engineering recovery, the Monitor view of the Administrator tool does not display summary statistics for recovered jobs if the Data Integration Service stops after the application is submitted to the cluster.
Midstream parsing complex functions PARSE_JSON and PARSE_XML cannot parse the data if the JSON keys or XML tags have a period or other special characters.
Remove special characters in the JSON keys or XML tags.
The Developer tool accepts more than two arguments in the PARSE_JSON and PARSE_XML complex functions.
Use two arguments with the PARSE_JSON and PARSE_XML complex functions.
When you compare two mappings containing Aggregator transformation that are different, the compare report shows the cacheDir value as object.
The infacmd dis queryDesignTimeObjects and queryRunTimeObjects commands fail when you assign the reserved keywords such as where, name, project, and folder as value in a query.
Don't assign the reserved keywords in a query.
When a mapping configured with HDP 3.1 on the Blaze engine writes to an external ORC target, the mapping stages data to a temporary Parquet table and uses HS2 to write to the target, which might cause performance impact due to additional staging tables.
The Developer tool accepts complex data type definition names with special characters, which cannot be used in a mapping. When you validate expressions with special characters in the name, parse errors occur.
Do not use special characters in complex data type definition names.
You cannot update the tag description in the following situations:
When an intelligent structure model is used midstream, Intelligent Structure Discovery passes only the first array element of each array in the input. All other array elements are discarded.
For example, if a "Companies" root element contains an array with multiple "Company" elements, Intelligent Structure Discovery passes only the first "Company" element.
When you run data preview on a complex file data object that contains column names with special characters, the Data Integration Service fails to pass the request to the Spark engine.