Install Python for the Python transformation on Hadoop

Install Python for the Python transformation on Hadoop

Install Jep for streaming mappings

Install Jep for streaming mappings

Streaming mappings require Jep as part of the Python installation. If you run the Python transformation in a streaming mapping, complete the following tasks to install Jep:

Set JAVA_HOME

Jep has a dependency on JDK. Before you install Jep, you must set the environment variable
$JAVA_HOME
. The Informatica domain contains an installation of Java, so you can use the JDK that is packaged in the Informatica domain.
Run the following command to set
$JAVA_HOME
to the packaged JDK:
export JAVA_HOME=$INFA_HOME/java

Install Jep

Run the following command to change directories to
$PYTHONHOME
:
cd $PYTHONHOME
Run the following command to install Jep:
bin/pip3 install jep

Copy Python to the Informatica domain

Run the following command to remove all files from the following directory in the Informatica domain:
rm -rf $INFA_HOME/services/shared/spark/python/*
Run the following command to copy the Python installation from
$PYTHONHOME
to the Informatica domain:
cp -r $PYTHONHOME/* $INFA_HOME/services/shared/spark/python/

Integrate Jep with the Hadoop connection

To integrate Jep with the Hadoop connection, complete the following tasks in the Administrator tool:
  1. Navigate to
    Manage > Connections
    .
  2. Select the Hadoop connection that connects to the Hadoop cluster.
  3. Edit the
    Spark configuration
    .
  4. Edit the
    Advanced Properties
    .
  5. Configure the following advanced properties:
    infaspark.pythontx.executorEnv.LD_PRELOAD
    Location of the Python shared library. For example, set the property to the following value:
    $INFA_HOME/services/shared/spark/python/lib/libpython3.6m.so
    infaspark.pythontx.executorEnv.PYTHONHOME
    Location of the Python installation directory. Keep the following default value:
    $INFA_HOME/services/shared/spark/python/
    infaspark.pythontx.submit.lib.JEP_HOME
    Location of the Jep package. For example, set the property to the following value:
    $INFA_HOME/services/shared/spark/python/lib/python3.6/site-packages/jep/

0 COMMENTS

We’d like to hear from you!