English
  • Data Engineering Streaming 10.2.2
  • All Products

Table of Contents

Search

  1. Preface
  2. Introduction to Big Data Streaming
  3. Big Data Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Connections
  9. Sample Files

User Guide

User Guide

Back Next

Streaming Process

Streaming Process

A streaming mapping receives data from unbounded data sources. An unbounded data source is one where data is continuously flowing in and there is no definite boundary. Sources stream data as events. The Spark engine processes the data and continuously updates the results to a result table.
The following image shows how the Spark engine receives data and publishes data in micro batches:
the spark engine receives data from a kafka source and publishes the data in batches to an hdfs target.
The Spark engine uses Spark Structured Streaming to process data that it receives in batches. Spark Structured Streaming receives data from streaming sources such as Kafka and divides the data into micro batches. The Spark engine continuously processes data streams as a series of small batch jobs with end-to-end latencies as low as 100 milliseconds with exactly-once fault tolerance guarantees.
For more information about Spark Structured Streaming, see the Apache Spark documentation at https://spark.apache.org/documentation.html.
You can perform the following high-level tasks in a streaming mapping:
  1. Identify sources from which you need to stream data. You can access data that is in XML, JSON, Avro, flat, or binary format.
    You can use Kafka Amazon Kinesis stream and Azure Event Hubs sources to connect to multiple big data sources.
  2. Configure the mapping and mapping logic to transform the data.
  3. Run the mapping on the Spark engine in the Hadoop environment.
  4. Write the data to Kafka targets, HDFS complex files, HBase, Azure Event Hubs, Azure Data Lake Store, JMS, and Kinesis Firehose delivery streams.
  5. Monitor the status of your processing jobs. You can view monitoring statistics for your processing jobs in the Monitoring tool.
    Support for monitoring jobs is deferred. Support will be reinstated in a future release.