Preface
Introduction to Informatica Edge Data Streaming
- Informatica Edge Data Streaming Overview
- Edge Data Streaming Architecture
- Data Flow Model
- Edge Data Streaming High-Level Process
- Edge Data Streaming Data Flow Process
- Edge Data Streaming User Interface
- Example
Licenses
- Licenses Overview
- Viewing the License Details
- Updating a License
- Removing a License
Using Informatica Administrator
- Using Informatica Administrator Overview
- Manage Tab - Domain View
- Manage Tab - Services and Nodes View
  - Domain
  - Application Services
- Logs Tab
- Security Tab
- Managing Your Account
- Logging In
- Password Management
  - Changing Your Password
- Managing Users and Groups
  - Default Administrator
- Managing Users
  - Creating Users
  - Unlocking a User Account
- Managing Groups
  - Adding a Native Group
- Managing Privileges
  - Domain Privileges
- Roles
- Managing Roles
- Usage Collection Policy
  - Disabling Informatica Data Usage
Creating and Managing the Edge Data Streaming Service
- Creating and Managing the Edge Data Streaming Service Overview
- Creating the Edge Data Streaming Service
  - Creating the Edge Data Streaming Service in the Administrator Tool
  - Creating the Edge Data Streaming Service using Informatica Command Line Program
- Editing the Edge Data Streaming Service
Edge Data Streaming Entity Types
- Edge Data Streaming Entity Types Overview
- Aggregators
  - Aggregator Properties
- Built-in Source Service Types
- Built-in Target Service Types
- Built-in Transformation Types
- Using Parameters in Entity Properties
  - Setting Values for Parameters
  - Examples
- Custom Entity Types
- Advanced Configuration for Entities
- Configuring High Availability for Entities
Edge Data Streaming Nodes
- Edge Data Streaming Nodes Overview
- Node Groups
- Node Group Management Tab
- Working with Node Groups
Data Connections
- Data Connections Overview
- Ultra Messaging Data Connection
- WebSocket Data Connection
  - Configuring an External Load Balancer
  - WebSocket Data Connection Properties
Working With Data Flows
- Working With Data Flows Overview
- Types of Data Flows
- Creating a Data Flow
- Data Flow Design Tab
- Adding Entities to a Data Flow
- Edge Data Streaming Node Mapping
- Deploying a Data Flow
- Undeploying a Data Flow
- Undeploying and Deploying all Data Flows
  - Undeploying All Data Flows
  - Deploying All Data Flows
- Editing Data Flows and Entities
- Cloning a Data Flow
- Removing Data Flows and Entities
- Verifying Entity Properties
- Configuring Targets with Data Connection and Target Service Properties
  - Getting the Topic Name Assigned to a Connection
  - Getting the Receiver Type ID of a Target Service
- Getting Entity Alerts
Managing the Edge Data Streaming Components
- Managing the Edge Data Streaming Components Overview
- Administrator Daemon Management
- Edge Data Streaming Node Management
- Managing the Informatica Domain
Security
- Security Overview
- Authentication
- Component Security
- Secure Communication Within the Components
- Secure Data Storage
  - Update Encryption Keys
  - Updating Security Keys
- Secure Source Services and Target Services
- Privileges and Roles
High Availability
- High Availability Overview
- Restart and Failover
- Resilience
- Configuring High Availability in Edge Data Streaming
  - Design-Time High Availability
  - Run-Time High Availability
    - Entity High Availability
Disaster Recovery
- Disaster Recovery Overview
- Step 1: Replicate the EDS Installation
- Step 2: Back Up Data Flows
- Step 3: Back Up The Node Groups
- Step 4: Set Parameters
- Step 5: Replicate Source Files and Position Files
- Step 6: Restore EDS from the Disaster Recovery Site
Monitoring Edge Data Streaming Entities
- Monitoring Edge Data Streaming Entities Overview
- Viewing the Monitoring Tab
- Monitoring Tab Layout
  - System View
  - Grid View
- Edge Data Streaming Statistics
Appendix A: Troubleshooting
- Troubleshooting Licenses
- Troubleshooting Edge Data Streaming Node Issues
- Troubleshooting Administrator Daemon Issues
- Troubleshooting the Administrator Tool
- Troubleshooting Apache ZooKeeper
- Troubleshooting Component Connectivity Issues
- Troubleshooting Edge Data Streaming High Availability
- Troubleshooting Data Flows
- Troubleshooting Entities
- Troubleshooting Monitoring Tab Views
Appendix B: Frequently Asked Questions
- Frequently Asked Questions About Edge Data Streaming
Appendix C: Regular Expressions
Appendix D: Command Line Program
- Command Line Program Overview
- infacmd eds Plugin
- Running Commands
- infacmd Return Codes
- infacmd eds Command Reference
Appendix E: Configuring Edge Data Streaming to Work With a ZooKeeper Observer
Appendix F: Glossary
- Administrator Daemon
- data flow
- Informatica Administrator
- receiver type ID
- source service
- target service
- topic resolution domain
- unicast topic resolution daemon (LBMRD)
- EDS Node

User Guide

2.4.0
- 2.5.0

Back Next

Edge Data Streaming Data Flow Process

Edge Data Streaming
Data Flow Process

You use the

Administrator tool

to design the flow of data from the data source to the data target and to deploy the data flow.

The

Administrator Daemon

pushes the data flow configuration information to Apache ZooKeeper. The

EDS Node

s download the configuration information and start the source services and target services that the configuration specifies. Source services read data in blocks and publish messages through a data connection. Target services receive the data and write the data to a data target. The EDS Node monitors the entities in the data flow and sends information about state and statistics to the

Administrator Daemon

. The

Administrator Daemon

sends this information to the

Administrator tool

For example, an application writes log data to log files in the following directory:

/usr/app/logs/

. You want to transfer the data contained in the log files to an HDFS cluster. To transfer the data, install

EDS Node

s on the application host machine and target host machine. As part of performing post-installation tasks, start a

EDS Node

Node1 on the application host and a

EDS Node

Node2 on the target host.

The following image shows how

EDS

works:

The image shows the sequence of operations in Edge Data Streaming.

The image numbers the operations in the order of occurrence. The following steps describe the sequence of operations:

Use the

Administrator tool

to create and deploy a data flow. When you configure the data connection in the data flow, use the Ultra Messaging or a WebSockets data connection. In the data flow, create a source service. Specify the source directory as

/usr/app/logs/

, and map the service to Node1. Create an HDFS target service and map the target service to Node2. Connect the source service to the target service, and add any transformations that you want to apply to the data. Finally, deploy the data flow. The

Administrator Daemon

sends the data flow configuration information to ZooKeeper.

The EDS Nodes download data flow configuration information from ZooKeeper. The

EDS Node

Node1 starts a source service. Similarly, Node2 starts a target service.

The source service reads data from the source files and publishes that data as messages on a topic.

EDS

applies the transformations that you added to the data flow. The target service subscribes to the topic, receives the data, and writes it to the HDFS cluster.

The EDS Node sends information about state and statistics to the

Administrator Daemon

. The

Administrator Daemon

publishes the information through the Edge Data Streaming Service. You can view the information on the

Monitoring

tab in the

Administrator tool

Introduction to Informatica Edge Data Streaming

Creating a Data Flow

Download Guide

Watch