The following image shows how the Spark engine receives data and publishes data in micro batches:

The Spark engine uses Spark Structured Streaming to process data that it receives in batches. Spark Structured Streaming receives data from streaming sources such as Kafka and divides the data into micro batches. The Spark engine continuously processes data streams as a series of small batch jobs with end-to-end latencies as low as 100 milliseconds with exactly-once fault tolerance guarantees.

For more information about Spark Structured Streaming, see the Apache Spark documentation at https://spark.apache.org/documentation.html.

You can perform the following high-level tasks in a streaming mapping:

Identify sources from which you need to stream data. You can access data that is in XML, JSON, Avro, flat, or binary format.
In Hadoop environment, you can use Kafka Amazon Kinesis stream and Azure Event Hubs sources to connect to multiple data engineering sources.
Configure the mapping and mapping logic to transform the data.
Run the mapping on the Spark engine in the Hadoop environment or on the Databricks Spark Engine in the Databricks environment.
Write the data to Kafka targets, HDFS complex files, HBase, Azure Event Hubs, Amazon S3, Azure Data Lake Storage, JMS, and Kinesis Firehose delivery streams.
Monitor the status of your processing jobs. You can view monitoring statistics for your processing jobs in the Monitoring tool.