Table of Contents

Search

  1. Preface
  2. Introduction to Big Data Management Administration
  3. Authentication
  4. Running Mappings on a Cluster with Kerberos Authentication
  5. Authorization
  6. Cluster Configuration
  7. Cloud Provisioning Configuration
  8. Data Integration Service Processing
  9. Connections
  10. Multiple Blaze Instances on a Cluster
  11. Monitoring REST API

Big Data Management Administrator Guide

Big Data Management Administrator Guide

Overview

Overview

When you use the Blaze engine to run mappings, Blaze uses a Grid Manager at run time to allot tasks to various nodes in a Hadoop cluster. The Grid Manager aids in resource allocation.
You can use the same Hadoop cluster to stage your test environment and establish a production environment. To control resource use on the cluster, you can establish a separate Blaze instance for testing and another for production.
Each instance requires a separate Grid Manager. You create an additional Grid Manager by performing a series of steps to create separate infrastructure for each Blaze instance, including a unique namespace and a Hadoop connection for each Blaze instance to use. Each Blaze instance also requires a separate Hadoop connection.
The following image shows how a separate Data Integration Service on the domain creates a separate Grid Manager on the cluster:
The image shows a flow diagram depicting a Test and a Production Data Integration Service on the Domain. On the cluster, each Data Integration Service starts a separate Grid Manager, which communicate with a single Resource Manager to send data to the Node Manager.
The image shows how separate Data Integration Services use separate Blaze instances. Each instance uses a separate Grid Manager to communicate with the cluster resource manager to balance resources.
Perform the following steps to set up separate Blaze instances:

    Step 1. Prepare the Hadoop cluster for the Blaze engine.

    Step 2. Configure Data Integration Service properties.

    Step 3. Create a new Hadoop connection.

    Step 4. Configure Additional Hadoop Connection Properties.

    Step 5. Set Mapping Preferences.


0 COMMENTS

We’d like to hear from you!