Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Data Engineering Integration
  3. Mappings
  4. Mapping Optimization
  5. Sources
  6. Targets
  7. Transformations
  8. Python Transformation
  9. Data Preview
  10. Cluster Workflows
  11. Profiles
  12. Monitoring
  13. Hierarchical Data Processing
  14. Hierarchical Data Processing Configuration
  15. Hierarchical Data Processing with Schema Changes
  16. Intelligent Structure Models
  17. Blockchain
  18. Stateful Computing
  19. Appendix A: Connections Reference
  20. Appendix B: Data Type Reference
  21. Appendix C: Function Reference

Advanced Properties for the Blaze Engine

Advanced Properties for the Blaze Engine

If you want to use the Blaze engine to run mappings on the cloud platform cluster, you must set cluster configuration properties in the Software Setting property of the Create Cluster task.
Configure the Create Cluster task to set configuration properties in *-site.xml files on the cluster. Hadoop clusters run based on these settings.
Use a text file to specify *-site.xml file settings, and specify the path to the file in the Software Settings property.
The following text shows sample configuration of the Software Settings property file:
[ { "Classification":"yarn-site", "Properties":{ "yarn.scheduler.minimum-allocation-mb":"250", "yarn.scheduler.maximum-allocation-mb":"8192", "yarn.nodemanager.resource.memory-mb":"16000", "yarn.nodemanager.resource.cpu-vcores":"12" } }, { "Classification":"core-site", "Properties":{ "hadoop.proxyuser.<DIS/OSPUSER>.groups":"<group names>", "hadoop.proxyuser.<DIS/OSPUSER>.hosts":"*" } } ]

yarn-site

yarn.scheduler.minimum-allocation-mb
The minimum RAM available for each container. Required for Blaze engine resource allocation.
yarn.scheduler.maximum-allocation-mb
The maximum RAM available for each container. Required for Blaze engine resource allocation.
yarn.nodemanager.resource.memory-mb
The maximum RAM available for each container. Set the maximum memory on the cluster to increase resource memory available to the Blaze engine.
yarn.nodemanager.resource.cpu-vcores
The number of virtual cores for each container. Required for Blaze engine resource allocation.

core-site

hadoop.proxyuser.<proxy user>.groups
Defines the groups that the proxy user account can impersonate. On a secure cluster the <proxy user> is the Service Principal Name that corresponds to the cluster keytab file. On a non-secure cluster, the <proxy user> is the system user that runs the Informatica daemon.
Set to group names of impersonation users separated by commas. If less security is preferred, use the wildcard " * " to allow impersonation from any group.
hadoop.proxyuser.<user>.hosts
Defines the host machines from which impersonated connections are allowed. On a secure cluster the <proxy user> is the Service Principal Name that corresponds to the cluster keytab file. On a non-secure cluster, the <proxy user> is the system user that runs the Informatica daemon.
Set the property to " * " to allow impersonation from any host. This is required to run a Blaze mapping on a cloud platform cluster.

0 COMMENTS

We’d like to hear from you!