Table of Contents

Search

  1. Preface
  2. Introduction to Big Data Streaming
  3. Big Data Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Connections
  9. Sample Files

Example

Example

You run the IT department of a major bank that has millions of customers. You want to monitor network activity in real time. You need to collect network activity data from various sources such as firewalls or network devices to improve security and prevent attacks. The network activity data includes Denial of Service (DoS) attacks and failed login attempts made by customers. The network activity data is written to Kafka queues.
You perform the following tasks:
  1. Create a streaming mapping to read data from the Kafka queues that stream data in JSON, XML, CSV, or Avro formats.
  2. Add a Lookup transformation to get data from a particular customer ID.
  3. Add a Window transformation to accumulate the streamed data into data groups before processing the data.
  4. Process the data. Add an Aggregator transformation to perform aggregations on the data from the customer ID.
  5. Monitor jobs. Monitor statistics for the mapping job on the Monitoring tab of the Administrator tool.
    Support for monitoring jobs is deferred. Support will be reinstated in a future release.
The following image shows the mapping:
The mapping example shows a Kafka input, a Lookup transformation, a Window transformation, an Aggregator transformation, and a Kafka output.

0 COMMENTS

We’d like to hear from you!