Table of Contents


  1. Preface
  2. Introduction to Data Engineering Streaming
  3. Data Engineering Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Transformation in Streaming Mappings
  8. Window Transformation
  9. Appendix A: Connections
  10. Appendix B: Monitoring REST API Reference
  11. Appendix C: Sample Files



You run the IT department of a major bank that has millions of customers. You want to monitor network activity in real time. You need to collect network activity data from various sources such as firewalls or network devices to improve security and prevent attacks. The network activity data includes Denial of Service (DoS) attacks and failed login attempts made by customers. The network activity data is written to Kafka queues.
You perform the following tasks:
  1. Create a streaming mapping to read data from the Kafka queues that stream data in JSON, XML, CSV, or Avro formats.
  2. Add a Lookup transformation to get data from a particular customer ID.
  3. Add a Window transformation to accumulate the streamed data into data groups before processing the data.
  4. Process the data. Add an Aggregator transformation to perform aggregations on the data from the customer ID.
  5. Monitor jobs. Monitor statistics for the mapping job on the Monitoring tab of the Administrator tool.
The following image shows the mapping:
The mapping example shows a Kafka input, a Lookup transformation, a Window transformation, an Aggregator transformation, and a Kafka output.


We’d like to hear from you!