Configuring the Messaging Mode in Vibe Data Stream for Machine Data

Configuring the Messaging Mode in Vibe Data Stream for Machine Data

Load Balancing

Load Balancing

You can configure the VDS data flows to use the Load Balancing message mode to reduce the load on a target service. To reduce the load on a target service, you can deploy the target service on multiple nodes, and you can configure the target service instances to write data to the target concurrently. VDS detects the presence of multiple instances of a target service and balances the load across the instances.
A target service sends a source service an acknowledgment after the target service processes a message. When a data flow contains multiple target services, a source service stops publishing messages when the number of unacknowledged messages reaches 1,000. A source service identifies a message as acknowledged only if it receives an acknowledgment from all the target services that receive the message.
When you deploy a target service on multiple
VDS Node
s,
VDS
uses the round-robin method to distribute the messages across the instances.
VDS
avoids data duplication within the load balanced group of instances by delivering each message to one instance. Deploying a target service on multiple nodes ensures that a large volume of data from multiple sources does not overwhelm a single target service instance.
The following image shows how
VDS
balances the load across HDFS target services that you deployed on three nodes:
The data flow has one source service that publishes data as messages 1,2, and 3 on a topic called logs. Three instances of an HDFS target service receive the messages.
VDS
balances the load as follows:
  1. A source service reads data from the data source and publishes it as three messages over a topic called logs.
  2. An HDFS target service instance runs on three nodes.
  3. You have deployed the target service on three nodes for purposes of load balancing.
    VDS
    balances the load across the three instances of the target service in round-robin fashion.
The distribution of messages depends on the load on individual instances. Although
VDS
uses the round-robin method, some instances might receive more messages than other instances.

0 COMMENTS

We’d like to hear from you!