Hadoop Files V2 Connector

Hadoop Files V2 Connector

Access Kerberos-enabled Hadoop cluster

Access Kerberos-enabled Hadoop cluster

Configure the
/etc/hosts
file and copy the Kerberos configuration file for HDFS instances that use Kerberos authentication.
  1. Open the
    /etc/hosts
    file located in the
    /etc
    directory on the Secure Agent machine on Linux.
  2. To configure the Secure Agent to work with the Kerberos Key Distribution Center (KDC), make an entry of the KDC hosts in the
    /etc/hosts
    file.
  3. Copy the
    krb5.conf
    configuration file from the
    /etc
    directory in the Hadoop cluster node to the following location:
    <Secure Agent installation directory>/apps/jdk/zulu<latest_version>/jre/lib/security
    If the Secure Agent is already installed, copy to:
    <Secure Agent installation directory>/apps/jdk/jre/lib/security
  4. If the cluster is SSL enabled, import the certificate alias file to the following location:
    <Secure Agent installation directory>/jdk/jre/lib/security/cacerts
    If the Secure Agent is already installed, import to:
    <Secure Agent installation directory>/apps/jdk/zulu<latest_version>/jre/lib/security/cacerts
  5. Restart the Secure Agent.

0 COMMENTS

We’d like to hear from you!
Nitin N - January 12, 2023

> path in point 3 is incorrect  /jdk/jre/lib/security

it should be  /apps/jdk/......

 

> this guide should also document what all ports are needs to connect to hadoop cluster, there are plenty of ports and its unclear which one exactly get used to open in the firewall. here is the sample:

Datanode ports
dfs.datanode.ipc.address  9867
dfs.datanode.address  1004
dfs.datanode.http.address 1006
dfs.datanode.https.address  9865
dfs.journalnode.rpc-address 8485
dfs.journalnode.http-address 8480
dfs.journalnode.https-address 8481

 


Namendoe ports
fs.defaultFS  8020
dfs.namenode.servicerpc-address 8022
dfs.namenode.http-address 9870
dfs.namenode.secondary.http-address 9868
dfs.secondary.https.port 9869
dfs.https.port  9871

 

      Informatica Documentation Team - January 13, 2023

      Hi Nitin N,

      We’re working to address your comments and will get back to you.

      Thanks,

      Informatica Documentation team


      Informatica Documentation Team - January 19, 2023

      Hi Nitin N,

      We’re happy to let you know that we've addressed your feedback in the documentation. 

      If the Secure Agent is already installed, you need to copy the certificate to the following directory: /apps/jdk/.

      To connect to the Hadoop cluster, you need to use the fs.defaultFS  namenode port. You can see the updated information in the NameNode URI description field on the following page: Hadoop Files V2 connection properties.

      Thanks,

      Informatica Documentation team


      Nitin N - January 20, 2023

      none of my feedback is correctly addressed

      1> all the java path is still incorrect in this page, work with the right tech folks to provide the right info in the document

      2> namenode port alone is not enough for this connector to work, we may specify the name node port in connection properties but it also needs datanode connectivity. Thats the missing piece of info. hence the feedback to mention what all ports of datanode and namenode is required.

      [for ex: if its a oracle, we know there will be only 1 port (1433 by default) in use and we can easily open the firewall. But for a hadoop cluster there will be big list of ports that i already mentioned below, question is what all ports are needed for this connector to work]

      Informatica Documentation Team - January 24, 2023

      Hi Nitin N,

      The port numbers might change based on the distribution vendors. To prevent issues related to port numbers, it is recommended to enable all the required ports for the following services: datanode, namenode, KMS, and ZooKeeper.

      Even after enabling all the required ports, if you still face issues with the port numbers, we request you to raise a ticket with Global Customer Support. 

      Regards,

      Informatica Documentation Team