What's New and Changed (10.5.2)

Back Next

Python Transformation

If you upgrade to version 10.2.2, the Python transformation can process data more efficiently in Big Data Management.

To experience the improvements in performance, configure the following Spark advanced properties in the Hadoop connection:

infaspark.pythontx.exec: Required to run a Python transformation on the Spark engine for Data Engineering Integration. The location of the Python executable binary on the worker nodes in the Hadoop cluster.

For example, set to:
infaspark.pythontx.exec=/usr/bin/python3.4

If you use the installation of Python on the Data Integration Service machine, set the value to the Python executable binary in the Informatica installation directory on the Data Integration Service machine.

For example, set to:
infaspark.pythontx.exec=INFA_HOME/services/shared/spark/python/lib/python3.4
infaspark.pythontx.executorEnv.PYTHONHOME: Required to run a Python transformation on the Spark engine for Data Engineering Integration and Data Engineering Streaming. The location of the Python installation directory on the worker nodes in the Hadoop cluster.

For example, set to:
infaspark.pythontx.executorEnv.PYTHONHOME=/usr

If you use the installation of Python on the Data Integration Service machine, use the location of the Python installation directory on the Data Integration Service machine.

For example, set to:
infaspark.pythontx.executorEnv.PYTHONHOME= INFA_HOME/services/shared/spark/python/

After you configure the advanced properties, the Spark engine does not use Jep to run Python code in the Python transformation.