Communities
A collaborative platform to connect and grow with like-minded Informaticans across the globe
Product Communities
Connect and collaborate with Informatica experts and champions
Discussions
Have a question? Start a Discussion and get immediate answers you are looking for
User Groups
Customer-organized groups that meet online and in-person. Join today to network, share ideas, and get tips on how to get the most out of Informatica
Get Started
Community Guidelines
Knowledge Center
Troubleshooting documents, product guides, how to videos, best practices, and more
Knowledge Base
One-stop self-service portal for solutions, FAQs, Whitepapers, How Tos, Videos, and more
Support TV
Video channel for step-by-step instructions to use our products, best practices, troubleshooting tips, and much more
Documentation
Information library of the latest product documents
Velocity (Best Practices)
Best practices and use cases from the Implementation team
Learn
Rich resources to help you leverage full capabilities of our products
Trainings
Role-based training programs for the best ROI
Certifications
Get certified on Informatica products. Free, Foundation, or Professional
Product Learning Paths
Free and unlimited modules based on your expertise level and journey
Resources
Library of content to help you leverage the best of Informatica products
Tech Tuesdays Webinars
Most popular webinars on product architecture, best practices, and more
Product Availability Matrix
Product Availability Matrix statements of Informatica products
SupportFlash
Monthly support newsletter
Support Documents
Informatica Support Guide and Statements, Quick Start Guides, and Cloud Product Description Schedule
Product Lifecycle
End of Life statements of Informatica products
Ideas
Events
Change Request Tracking
Marketplace
English
  • Data Engineering Integration 10.2.2 Service Pack 1
  • All Products

Table of Contents

Search

  1. Preface
  2. Part 1: Hadoop Integration
  3. Part 2: Databricks Integration
  4. Appendix A: Connections

Integration Guide

Integration Guide

Back Next

Prepare a Python Installation

Prepare a Python Installation

If you want to use the Python transformation, you must ensure that the worker nodes on the Hadoop cluster contain an installation of Python. You must complete different tasks depending on the product that you use.

Installing Python for Big Data Management

To use the Python transformation in Big Data Management, the worker nodes on the cluster must contain a uniform installation of Python. You can ensure that the installation is uniform in one of the following ways:
  • Verify that all worker nodes on the cluster contain an installation of Python in the same directory, such as
    /usr/lib/python
    , and that each Python installation contains all required modules. You do not re-install Python, but you must reconfigure the following Spark advanced property in the Hadoop connection: 
    infaspark.pythontx.executorEnv.PYTHONHOME
  • Install Python on every Data Integration Service machine. You can create a custom installation of Python that contains specific modules that you can reference in the Python code. When you run mappings, the Python installation is propagated to the worker nodes on the cluster.
If you choose to install Python on the Data Integration Service machines, complete the following tasks:
  1. Install Python.
  2. Optionally, install any third-party libraries such as numpy, scikit-learn, and cv2. You can access the third-party libraries in the Python transformation.
  3. Copy the Python installation folder to the following location on the Data Integration Service machine: 
    <Informatica installation directory>/services/shared/spark/python
    If the Data Integration Service machine already contains an installation of Python, you can copy the existing Python installation to the above location.
Changes take effect after you recycle the Data Integration Service.

Installing Python for Big Data Streaming

To use the Python transformation in Big Data Streaming, you must install Python and the Jep package. Because you must install Jep, the Python version that you use must be compatible with Jep. You can use one of the following versions of Python:

    2.7

    3.3

    3.4

    3.5

    3.6

To install Python and Jep, complete the following tasks:
  1. Install Python with the
    --enable-shared
     option to ensure that shared libraries are accessible by Jep.
  2. Install Jep. To install Jep, consider the following installation options:
    • Run
      pip install jep
      . Use this option if Python is installed with the pip package.
    • Configure the Jep binaries. Ensure that
      jep.jar
       can be accessed by Java classloaders, the shared Jep library can be accessed by Java, and Jep Python files can be accessed by Python.
  3. Optionally, install any third-party libraries such as numpy, scikit-learn, and cv2. You can access the third-party libraries in the Python transformation.
  4. Copy the Python installation folder to the following location on the Data Integration Service machine: 
    <Informatica installation directory>/services/shared/spark/python
    If the Data Integration Service machine already contains an installation of Python, you can copy the existing Python installation to the above location.
Changes take effect after you recycle the Data Integration Service.
0 COMMENTS
We’d like to hear from you!