Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Big Data Management
  3. Mappings
  4. Sources
  5. Targets
  6. Transformations
  7. Data Preview
  8. Cluster Workflows
  9. Profiles
  10. Monitoring
  11. Hierarchical Data Processing
  12. Hierarchical Data Processing Configuration
  13. Hierarchical Data Processing with Schema Changes
  14. Intelligent Structure Models
  15. Stateful Computing
  16. Appendix A: Connections
  17. Appendix B: Data Type Reference
  18. Appendix C: Function Reference

Advanced Properties for the Blaze Engine

Advanced Properties for the Blaze Engine

If you want to use the Blaze engine to run mappings on the cloud platform cluster, you must set cluster configuration properties in the Software Setting property of the Create Cluster task.
Configure the Create Cluster task to set configuration properties in *-site.xml files on the cluster. Hadoop clusters run based on these settings.
Use a text file to specify *-site.xml file settings, and specify the path to the file in the Software Settings property.
The following text shows sample configuration of the Software Settings property file:
[ { "Classification":"yarn-site", "Properties":{ "yarn.scheduler.minimum-allocation-mb":"250", "yarn.scheduler.maximum-allocation-mb":"8192", "yarn.nodemanager.resource.memory-mb":"16000", "yarn.nodemanager.resource.cpu-vcores":"12" } }, { "Classification":"core-site", "Properties":{ "hadoop.proxyuser.<DIS/OSPUSER>.groups":"<group names>", "hadoop.proxyuser.<DIS/OSPUSER>.hosts":"*" } } ]

yarn-site

yarn.scheduler.minimum-allocation-mb
The minimum RAM available for each container. Required for Blaze engine resource allocation.
yarn.scheduler.maximum-allocation-mb
The maximum RAM available for each container. Required for Blaze engine resource allocation.
yarn.nodemanager.resource.memory-mb
The maximum RAM available for each container. Set the maximum memory on the cluster to increase resource memory available to the Blaze engine.
yarn.nodemanager.resource.cpu-vcores
The number of virtual cores for each container. Required for Blaze engine resource allocation.

core-site

hadoop.proxyuser.<proxy user>.groups
Defines the groups that the proxy user account can impersonate. On a secure cluster the <proxy user> is the Service Principal Name that corresponds to the cluster keytab file. On a non-secure cluster, the <proxy user> is the system user that runs the Informatica daemon.
Set to group names of impersonation users separated by commas. If less security is preferred, use the wildcard " * " to allow impersonation from any group.
hadoop.proxyuser.<user>.hosts
Defines the host machines from which impersonated connections are allowed. On a secure cluster the <proxy user> is the Service Principal Name that corresponds to the cluster keytab file. On a non-secure cluster, the <proxy user> is the system user that runs the Informatica daemon.
Set the property to " * " to allow impersonation from any host. This is required to run a Blaze mapping on a cloud platform cluster.

0 COMMENTS

We’d like to hear from you!