Table of Contents

Search

  1. Preface
  2. Introduction to Data Engineering Streaming
  3. Data Engineering Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Appendix A: Connections
  9. Appendix B: Monitoring REST API Reference
  10. Appendix C: Sample Files

Sources in a Streaming Mapping on Hadoop and Databricks

Sources in a Streaming Mapping on Hadoop and Databricks

A streaming mapping that runs in the Hadoop environment can include file, database, and streaming sources. A streaming mapping that runs in the Databricks environment can include streaming sources.
The following table lists the physical data objects that you can create based on the type of source you read from in either Hadoop or Databricks environment:
Sources in a Streaming Mapping
Environment
Amazon Kinesis
Hadoop, Databricks
Azure Event Hubs
Hadoop, Databricks
Confluent Kafka
Hadoop, Databricks
Google PubSub
Hadoop
HBase
Hadoop
JMS
Hadoop
Kafka
Hadoop, Databricks
MapR Streams
Hadoop
You can run streaming mappings in the Databricks environment on the AWS or Azure platforms. The following table shows the list of sources that you can include in a streaming mapping based on the cloud platform:
Sources
Cloud Platform
Amazon Kinesis
AWS
Azure Event Hubs
Azure
Confluent Kafka
AWS, Azure
Kafka
AWS, Azure