Big Data Management User Guide

Back Next

Configure the Create Cluster Task to Run Mappings on the Blaze Engine

If you want to use the Blaze engine to run mappings on the cloud platform cluster, you must set cluster configuration properties in the Software Setting property of the Create Cluster task.

Configure the Create Cluster task to set configuration properties in *-site.xml files on the cluster. Hadoop clusters run based on these settings.

The following text shows sample configuration of the Software Settings property:

[
  {
     "Classification":"yarn-site",
     "Properties":{
     "yarn.scheduler.minimum-allocation-mb":"250",
     "yarn.scheduler.maximum-allocation-mb":"8192",
     "yarn.nodemanager.resource.memory-mb":"16000",
     "yarn.nodemanager.resource.cpu-vcores":"12"
   }
  },
  {
    "Classification":"core-site",
    "Properties":{
    "hadoop.proxyuser.<DIS/OSPUSER>.groups":"<group names>",
    "hadoop.proxyuser.<DIS/OSPUSER>.hosts":"*"
  }
 }
]

yarn-site

yarn.scheduler.minimum-allocation-mb: The minimum RAM available for each container. Required for Blaze engine resource allocation.
yarn.scheduler.maximum-allocation-mb: The maximum RAM available for each container. Required for Blaze engine resource allocation.
yarn.nodemanager.resource.memory-mb: The maximum RAM available for each container. Set the maximum memory on the cluster to increase resource memory available to the Blaze engine.
yarn.nodemanager.resource.cpu-vcores: The number of virtual cores for each container. Required for Blaze engine resource allocation.

core-site

hadoop.proxyuser.<proxy user>.groups: Defines the groups that the proxy user account can impersonate. On a secure cluster the <proxy user> is the Service Principal Name that corresponds to the cluster keytab file. On a non-secure cluster, the <proxy user> is the system user that runs the Informatica daemon.; Set to group names of impersonation users separated by commas. If less security is preferred, use the wildcard " * " to allow impersonation from any group.
hadlop.proxyuser.<user>.hosts: Defines the host machines that a user account can impersonate. On a secure cluster the <proxy user> is the Service Principal Name that corresponds to the cluster keytab file. On a non-secure cluster, the <proxy user> is the system user that runs the Informatica daemon.; Set the property to " * " to allow impersonation from any host. This is required to run a Blaze mapping on a cloud platform cluster.

Rename Saved Search

Table of Contents

Big Data Management User Guide

Big Data Management User Guide

Configure the Create Cluster Task to Run Mappings on the Blaze Engine

Configure the Create Cluster Task to Run Mappings on the Blaze Engine

yarn-site

core-site