English
  • Data Engineering Streaming 10.2
  • All Products

Table of Contents

Search

  1. Preface
  2. Introduction to Intelligent Streaming
  3. Intelligent Streaming Configuration
  4. Connections
  5. Sources and Targets in a Streaming Mapping
  6. Intelligent Streaming Mappings
  7. Window Transformation
  8. Data Type Reference

Big Data Streaming User Guide

Big Data Streaming User Guide

Back Next

Troubleshooting Streaming Mappings with Lookup Transformation

Troubleshooting Streaming Mappings with Lookup Transformation

I use Sqoop as a Lookup transformation in the streaming mapping. The mapping fails, and I see the following error in the application logs of the CDH cluster: 
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
This error occurs if the MapReduce directory is configured incorrectly.
To resolve this error, perform the following steps:
  1. In the Yarn configuration, find the
    NodeManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml
     property.
  2. Add the following xml snippet: 
    <property> <name>mapreduce.application.classpath</name> <value>$HADOOP_MAPRED_HOME/,$HADOOP_MAPRED_HOME/lib/,
 $MR2_CLASSPATH</value> </property>
  3. Restart the affected services as indicated by Cloudera Manager and run the mapping again.
I use Sqoop as a Lookup transformation in the streaming mapping. The mapping validation fails, and I see the following errors in the Developer tool: 
Mapping1	Mapping	The transformation [output] contains a binary data type which you cannot use in a Streaming mapping. Use a valid data type. 

Mapping1 Mapping The transformation [output] contains a binary data type which you cannot use in a Streaming mapping. Use a valid data type. [ID:BINARY_FIELD_NOT_SUPPORTED_STREAMING]                
Lookup_ORACLE_TEST_CHAR    MRS/Sqoop_test/Mapping1

ORACLE_TEST_CHAR Relational Data Object  In relational column [TEST_NUMBER] with native datatype [decimal], the 
scale [-127] is not valid. [ID:INVALID_SCALE]  TEST_NUMBER MRS/Sqoop_test/ORACLE_TEST_CHAR
The errors occur if the Lookup transformation has a data type, such as binary, that Spark Streaming or Sqoop import does not support.
To resolve this error, delete the columns of the unsupported data type in the Lookup transformation and then validate the mapping.
For more information about data type support, see the
Informatica Big Data Management User Guide
.
I use Sqoop as a Lookup transformation in the streaming mapping. The mapping fails, and the following error appears in the application logs of the Hadoop cluster: 
User class threw exception: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/bin/../lib/hadoop-yarn/bin/yarn: 
line 318: /usr/java/default1/bin/java: No such file or directory
/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/bin/../lib/hadoop-yarn/bin/yarn: line 318: 
exec: /usr/java/default1/bin/java: cannot execute: No such file or directory
This error occurs when
jdk_home
 of the Hadoop distribution is configured incorrectly.
To resolve this error, perform the following steps:
  1. Find the
    hadoopEnv.properties
     in the following directory:
    <INFA_HOME>/services/shared/hadoop/<Hadoop distribution>/infaConf
  2. Set the
    jdk_home
     property correctly. For example, you can set the following value:
    infapdo.env.entry.hadoop_node_jdk_home=HADOOP_NODE_JDK_HOME=/usr/java/default
I use Sqoop as a Lookup transformation in a streaming mapping. The mapping fails, and I see the following error in the mapping logs: 
<INFA_HOME>/logs/node_automation/services/DataIntegrationService/disLogs/ms                                                                                                                                                           
:
Caused by: java.io.IOException: Cannot run program "<INFA_HOME>/services/shared/hadoop/<Hadoop distribution>/scripts/
HadoopFsRmRf" (in directory "."): error=13, Permission denied
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at java.lang.Runtime.exec(Runtime.java:620)
This error occurs when you do not have sufficient permissions on the
<Informatica installation directory>\externaljdbcjars
 directory in the Informatica domain. Get the required permissions and then run the mapping again.
For more information about the JDBC driver JAR files for Sqoop connectivity, see the
Informatica Big Data Management Cluster Integration Guide
.