Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Big Data Management
  3. Mappings in the Hadoop Environment
  4. Mapping Sources in the Hadoop Environment
  5. Mapping Targets in the Hadoop Environment
  6. Mapping Transformations in the Hadoop Environment
  7. Processing Hierarchical Data on the Spark Engine
  8. Configuring Transformations to Process Hierarchical Data
  9. Processing Unstructured and Semi-structured Data with an Intelligent Structure Model
  10. Stateful Computing on the Spark Engine
  11. Monitoring Mappings in the Hadoop Environment
  12. Mappings in the Native Environment
  13. Profiles
  14. Native Environment Optimization
  15. Cluster Workflows
  16. Connections
  17. Data Type Reference
  18. Function Reference
  19. Parameter Reference

Define YARN Queues

Define YARN Queues

You can define multiple YARN queues to redirect jobs to specific queues. You can redirect Blaze, Spark, Hive, and Sqoop jobs.
To redirect jobs on the Blaze engine to a specific queue, configure the following Blaze configuration property in the Hadoop connection:
Property
Description
YARN Queue Name
The YARN scheduler queue name used by the Blaze engine that specifies available resources on a cluster.
To redirect jobs on the Spark engine to a specific queue, configure the following Spark configuration property in the Hadoop connection:
Property
Description
YARN Queue Name
The YARN scheduler queue name used by the Spark engine that specifies available resources on a cluster. The name is case sensitive.
To redirect jobs on the Hive engine to a specific queue, configure the following Hive connection property:
Property
Description
Data Access Connection String
The Hive connection string to specify the queue name for Hive SQL override mappings on the Blaze engine.
Use the following format:
  • MapReduce.
    mapred.job.queue.name=<YARN queue name>
  • Tez.
    tez.queue.name=<YARN queue name>
For example,
jdbc:hive2://business.com:10000/default;principal=hive/_HOST@INFAKRB?mapred.job.queue.name-root.test
To redirect Sqoop jobs to a specific queue, configure the following JDBC connection property:
Property
Description
Sqoop Arguments
The Sqoop connection-level argument to direct a MapReduce job to a specific YARN queue.
Use the following format:
-Dmapred.job.queue.name=<YARN queue name>
If you do not direct the job to a specific queue, the Spark engine uses the default queue.


Updated October 23, 2019