Table of Contents

Search

  1. Preface
  2. Introduction to Hadoop Integration
  3. Before You Begin
  4. Amazon EMR Integration Tasks
  5. Azure HDInsight Integration Tasks
  6. Cloudera CDH Integration Tasks
  7. Hortonworks HDP Integration Tasks
  8. MapR Integration Tasks
  9. Appendix A: Connections

Create Blaze Engine Directories

Create Blaze Engine Directories

Create a blaze user account and directories required by the Blaze engine.
Complete the following tasks to prepare the Hadoop cluster for the Blaze engine:
Create a home directory for the blaze user.
If you created a blaze user, create home directory for the blaze user. For example,
hdfs hadoop fs -mkdir /user/blaze hdfs hadoop fs -chown blaze:blaze /user/blaze
If you did not create a blaze user, the Hadoop impersonation user is the default user.
Optionally, create a local services log directory.
By default, the Blaze engine writes the service logs to the YARN distributed cache. For example, run the following command:
mkdir -p /opt/informatica/blazeLogs
The Blaze configuration advanced properties in the Hadoop connection contains the following custom property for the services log directory:
infagrid.node.local.root.log.dir=$HADOOP_NODE_INFA_HOME/blazeLogs
$HADOOP_NODE_INFA_HOME gets set to the YARN distributed cache. If you create a directory, you must update the value of the custom property in the Hadoop connection.
Create an aggregated HDFS log directory.
Create a log directory on HDFS to contain aggregated logs for local services. For example:
hadoop fs -mkdir -p /var/log/hadoop-yarn/apps/informatica
The Blaze configuration advanced properties in the Hadoop connection contains the following custom property for the HDFS log directory:
infacal.hadoop.logs.directory=/var/log/hadoop-yarn/apps/informatica
Ensure that value of the custom property in the Hadoop connection matches the directory that you created.
Optionally, create a Blaze staging directory.
You can write the logs to the Informatica Hadoop staging directory, or you can create a Blaze staging directory. If you do not want to use the default location, create a staging directory on the HDFS. For example:
hadoop fs -mkdir -p /blaze/workdir
If you do not create a staging directory, clear the Blaze staging directory property value in the Hadoop connection and the Data Integration Service uses the HDFS directory
/tmp/blaze_<user name>
.
Grant permissions on the local services log directory, aggregated HDFS log directory, and the staging directory.
Grant permission to the following users:
  • Blaze user
  • Hadoop impersonation user
  • Mapping impersonation users
If the blaze user does not have permission, the Blaze engine uses a different user, based on the cluster security and the mapping impersonation configuration.

0 COMMENTS

We’d like to hear from you!