Table of Contents

Search

  1. Preface
  2. Introduction to Big Data Management Administration
  3. Authentication
  4. Running Mappings on a Cluster with Kerberos Authentication
  5. Authorization
  6. Cluster Configuration
  7. Cloud Provisioning Configuration
  8. Data Integration Service Processing
  9. Connections
  10. Multiple Blaze Instances on a Cluster
  11. Monitoring REST API

Big Data Management Administrator Guide

Big Data Management Administrator Guide

Distributed Queues

Distributed Queues

The Data Integration Service uses a distributed queue to store job information when big data recovery is enabled. The distributed queue is stored in the Model repository and is shared by all other nodes in the grid.
Any available node can run a job from the queue. If there is a long-running job on a node, another node can pick up jobs that are waiting to run.
Consider the following queueing process for distributed queues:
  1. A client submits a job request to the Data Integration Service, which stores job metadata in the distributed queue.
  2. When the Data Integration Service node has available resources, the Data Integration Service retrieves the job from the queue and sends it to the available node for processing.
  3. If a node fails while running a job, the job can fail over to another node. Any back-up node or node in the grid can take jobs from the queue.
  4. The interrupted job runs on the new node.
You can use the command
infacmd ms abortAllJobs
to abort all jobs in the queue, or
infacmd ms purgeDatabaseWorkTables
to clear the queue.

0 COMMENTS

We’d like to hear from you!