Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Edge Data Streaming
  3. Licenses
  4. Using Informatica Administrator
  5. Creating and Managing the Edge Data Streaming Service
  6. Edge Data Streaming Entity Types
  7. Edge Data Streaming Nodes
  8. Data Connections
  9. Working With Data Flows
  10. Managing the Edge Data Streaming Components
  11. Security
  12. High Availability
  13. Disaster Recovery
  14. Monitoring Edge Data Streaming Entities
  15. Appendix A: Troubleshooting
  16. Appendix B: Frequently Asked Questions
  17. Appendix C: Regular Expressions
  18. Appendix D: Command Line Program
  19. Appendix E: Configuring Edge Data Streaming to Work With a ZooKeeper Observer
  20. Appendix F: Glossary

User Guide

User Guide

Complex Data Flow with Data Duplication and Load Balancing

Complex Data Flow with Data Duplication and Load Balancing

A data flow can contain target services running on a single EDS Node or on multiple
EDS
Nodes. In such a deployment,
EDS
sends all the messages that the source publishes to each standalone target service. For a target service that you deploy on multiple nodes,
EDS
uses the round-robin method to distribute a subset of the messages to each target service instance.
The following image shows how
EDS
performs load balancing across multiple instances of a target service while duplicating data flows across different targets:
The data flow has a source service FileSourceSvc that publishes data as messages 1, 2 and 3 on a topic called logs. Two standalone target services on two EDS Nodes and three instances of an HDFC target service receive those messages. The HDFS target service HDFSSvc is deployed on three EDS Nodes for purposes of load balancing. EDS distributes messages in such a way that CassandraSvc and RulePointSvc receive all the messages that the source services publish. Simultaneously, EDS balances the load across the three instances of HDFSSvc in round-robin fashion and delivers a message to one instance of HDFSSvc.
The following process describes how EDS balances the load:
  1. A file source service FileSourceSvc publish messages on a topic called logs.
  2. A RulePoint target service named RulePointSvc, a standalone Cassandra target service named CassandraSvc, and three instances of an HDFS target service named HDFSSvc receive those messages.
  3. You deploy HDFSSvc on three EDS Nodes for purposes of load balancing.
  4. EDS
    distributes messages so that CassandraSvc and RulePointSvc receive all the messages that the source service publishes.
  5. Simultaneously,
    EDS
    performs load balancing across the three instances of HDFSClient in round-robin fashion.
    EDS
    delivers a message to one instance of HDFSClient.

0 COMMENTS

We’d like to hear from you!