Table of Contents

Search

  1. Abstract
  2. Installation and Upgrade
  3. 10.2 HF1 Fixed Limitations and Closed Enhancements
  4. 10.2 HF1 Known Limitations
  5. 10.2 Fixed Limitations and Closed Enhancements
  6. 10.2 Known Limitations
  7. Emergency Bug Fixes Merged into 10.2
  8. Informatica Global Customer Support

Big Data Known Limitations (10.2)

Big Data Known Limitations (10.2)

The following table describes known limitations:
Bug
Description
OCON-9377
When you configure Sqoop and run a Teradata Parallel Transporter mapping on a Cloudera cluster to export data of the Byte or Varbyte data type to a Teradata target, the mapping fails on the Blaze engine.
OCON-9376
If you configure Sqoop to export data of the Blob or Clob data type to a Teradata target, TDCH mappings fail on the Spark engine.
OCON-9143
In the read and write operations for a complex file data object, you cannot edit the precision and scale of elements within a field that is of a complex data type.
For example, if Field1 is of type array with string elements, you cannot edit the precision and scale of the string elements.
OCON-8850
If you configure Sqoop to export data of the Timestamp data type from a Hive source to a Microsoft Azure SQL Data Warehouse target, the mapping fails.
OCON-8779
If you configure Sqoop to export data of the Real data type to IBM DB2 z/OS targets, the mapping fails.
OCON-7521
Column profile run fails when the following conditions are true:
  1. You use the Cloudera Connector Powered by Teradata for Sqoop to read data from Teradata for a Sqoop data source.
  2. You create a column profile on the Sqoop data source.
  3. You run the profile using the Blaze engine.
Workaround: Create a separate Sqoop connection for each Sqoop data source with the
-split-by
option, and run a column profile on the data source.
OCON-7429
When you run a Teradata Parallel Transporter mapping on a Hortonworks cluster and on the Blaze engine to write data of the Byte or Varbyte data type to a Teradata target, the data gets corrupted. This issue occurs when you use the
hdp-connector-for-teradata-1.5.1.2.5.0.0-1245-distro.tar.gz
JAR.
Workaround: Use the
hdp-connector-for-teradata-1.4.1.2.3.2.0-2950-distro.tar.gz
JAR.
OCON-730
When you export data through Sqoop and there are primary key violations, the mapping fails and bad records are not written to the bad file. (456616)
OCON-7291
Mappings that read data from a Teradata source and contain the != (not equal) operator in the filter override query fail. This issue occurs if you run the Teradata Parallel Transporter mapping on a Hortonworks cluster and on the Blaze engine.
Workaround: Use a native expression with the ne operator instead of the != operator.
OCON-7280
If you configure Sqoop and update the columns in the advanced SQL query, the mapping fails on the Blaze engine.
OCON-7216
If a Sqoop source or target contains a column name with double quotes, the mapping fails on the Blaze engine. However, the Blaze Job Monitor incorrectly indicates that the mapping ran successfully and that rows were written into the target.
OCON-7212
If there are unconnected ports in a target, Sqoop mappings fail on the Blaze engine. This issue occurs when you run the Sqoop mapping on any cluster other than a Cloudera cluster.
Workaround: Before you run the mapping, create a table in the target database with columns corresponding to the connected ports.
OCON-7208
When you run a Sqoop mapping on the Blaze engine and the columns contain Unicode characters, the Sqoop program reads them as null values.
OCON-7205
When you run a Sqoop mapping on the Blaze engine to export data of the Numeric data type from Netezza, the scale part of the data is truncated.
OCON-7078
Sqoop mappings that import data from or export data to an SSL-enabled database fail on the Blaze engine.
OCON-7076
When you run a Sqoop mapping and abort the mapping from the Developer tool, the Sqoop map-reduce jobs continue to run.
Workaround: On the Sqoop data node, run the following command to kill the Sqoop map-reduce jobs:
yarn application -kill <application_ID>
OCON-688
When you enable Sqoop for a logical data object and export data to an IBM DB2 database, the Sqoop export command fails. However, the mapping runs successfully without any error. (456455)
OCON-471
When you enable Sqoop for a data object and a table or column name contains Unicode characters, the mapping fails. (452114)
LDM-3324
A column profile runs indefinitely when the following conditions are true:
  • The Hive data source for the profile resides on an Azure HDInsight cluster that uses WASB storage.
  • You create a column profile on the Hive source and enable data domain discovery.
  • You run the profile on the Hive engine in the Hadoop environment.
  • You do not use the JDBC connection as the profiling warehouse connection.
BDM-11049
After the Data Integration Service restarts, a mapping that contains a Data Processor transformation might fail to run on the Hive engine the first time that it executes.
BDM-9987
A mapping configured with a filter expression upstream of a Joiner transformation might be inconsistent with map-side joins.
BDM-8517
The infacmd ms RunMapping command does not return the job ID.
BDM-6754
When the Data Integration Service is configured to run with operating system profiles and you push the mapping to an HDInsight cluster with ADLS as storage, the mapping fails with the following error:
Exception Class: [java.lang.RuntimeException] Exception Message: [java.io.IOException: No FileSystem for scheme: adl]. java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: adl
BDM-6500
When you choose
Return All
for an unconnected lookup, mapping validation fails with the following unrelated error:
The port [{0}] in the transformation [{1}] contains an expression that is not valid. Error: Invalid unconnected transformation [{3}].
BDM-4597
A mapping with a joiner transformation that processes more than 4,294,967,294 rows in a single partition will fail.
Workaround: If possible, increase partitioning on the source.
BDM-3853
When the Blaze engine runs a mapping that uses source or target files in the WASB location on a cluster, the mapping fails with an error like:
java.lang.RuntimeException: [<error_code>] The Integration Service failed to run Hive query [exec0_query_6] for task [exec0] due to following error: <error_code> message [FAILED: ... Cannot run program "/usr/lib/python2.7/dist-packages/hdinsight_common/decrypt.sh": error=2, No such file or directory], ...
The mapping fails because the cluster attempts to decrypt the data but cannot find a file needed to perform the decryption operation.
Workaround: Find the following files on the cluster and copy them to the
/usr/lib/python2.7/dist-packages/hdinsight_common
directory on the machine that runs the Data Integration Service:
  • key_decryption_cert.prv
  • decrypt.sh
BDM-3687
When you run a Sqoop mapping on the Spark engine, the Sqoop map-reduce jobs run in the default yarn queue instead of the yarn queue that you configure.
Workaround: To run a map-reduce job in a particular yarn queue, configure the following property in the
Sqoop Arguments
field of the JDBC connection:
-Dmapreduce.job.queuename=<NameOfTheQueue>
To run a Spark job in a particular yarn queue, configure the following property in the Hadoop connection:
spark.yarn.queue=<NameOfTheQueue>
BDM-2222
The Spark engine does not run the footer row command configured for a flat file target. (459942)
BDM-2181
The summary and detail statistics empty for mappings run on Tez. (452224)
BDM-1271
If you define an SQL override in the Hive source and choose to update the output ports based on the custom query, the mapping fails on the Blaze engine.
BDM-10924
A mapping with an SQL query defined on a Hive source and target table, or with an SQL query generated on the Hive source, fails with an error like:
FAILED: ParseException line <line number> Failed to recognize predicate '<reserved keyword>'. Failed rule: 'identifier' in expression specification
Workaround: Create the following user-defined property in the hive-site.xml configuration set in the cluster configuration:
hive.support.sql11.reserved.keywords=false
BDM-10897
When you create or import type definition libraries using JSON objects with key names that have the dot operator (.), the following validation error occurs:
Evaluation failed and was not completed. Check the Developer tool logs for details.
BDM-10895
If the JSON object that you are reading from has a key name with a dot operator (.) and you use the key name in an expression, the following error occurs:
Cannot read complex definition
BDM-10859
A mapping that runs on the Hive engine fails when it meets the following conditions:
  • The mapping writes to multiple targets, and at least one is a Hive target.
  • The mapping contains a Router transformation.
  • Storage-based authorization is enabled on the mapping
BDM-10856
The Administrator tool fails to display a cluster configuration when one of the *-site.xml configuration sets contains a dot character (.) in the name, excluding the required character for the file extension.
BDM-10837
Mapping performance is slow when the Spark engine writes to a partitioned Hive table on Amazon S3.
BDM-10731
New mapping jobs fail when the Data Integration Service machine reaches the maximum configured memory.
BDM-10670
Memory leaks occur on the Data Integration Service machine when the Spark engine heap memory reaches 4 GB.
BDM-10570
The Spark job fails with out of memory errors when a mapping that converts relational data to hierarchical data contains more than three Aggregator and Joiner transformations.
Workaround: To convert relational data to a hierarchical data of more than four levels, develop more than one mapping to stage the intermediate data. For example, develop a mapping that converts relational data to a hierarchical data up to three levels. Use the hierarchical data in another mapping to generate a hierarchical data of four levels.
BDM-10566
Mappings fail when you use the impersonation user to read or write files within an HDFS encryption zone in a Hadoop cluster and operating system profiles are enabled for the Data Integration Service.
Workaround: Read and write the files outside of the encryption zone.
BDM-10098
Mappings with a Normalizer transformation fail on the Spark engine if the value in the Occurs column is large.


Updated June 29, 2020