Table of Contents

Search

  1. Preface
  2. Introduction to Data Engineering Streaming
  3. Data Engineering Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Appendix A: Connections
  9. Appendix B: Monitoring REST API Reference
  10. Appendix C: Sample Files

Sources in a Streaming Mapping on Databricks

Sources in a Streaming Mapping on Databricks

A streaming mapping that runs in the Databricks environment can include streaming sources.
Based on the type of source you read from, create the following data object:
Amazon Kinesis
A physical data object that represents data in a Amazon Kinesis Data Stream. After you create an Amazon Kinesis connection, create an Amazon Kinesis data object to read from Amazon Kinesis Data Streams.
Azure Event Hubs
A physical data object that represents data in Microsoft Azure Event Hubs data streaming platform and event ingestion service.
Confluent Kafka
A physical data object that represents data in a Kafka stream or a Confluent Kafka stream. After you configure a Messaging connection, create a Confluent Kafka data object to read data from Kafka brokers or from Confluent Kafka brokers using schema registry.
Kafka
A physical data object that represents data in a Kafka stream. After you configure a Messaging connection, create a Kafka data object to read from Apache Kafka brokers.
You can run streaming mappings in AWS Databricks service in AWS cloud ecosystems or in Azure Databricks service in Microsoft Azure cloud services. The following table shows the list of sources that you can include in a streaming mapping based on the Databricks service:
Sources
Services
Amazon Kinesis
AWS
Azure Event Hubs
Azure
Confluent Kafka
AWS, Azure
Kafka
AWS, Azure