Table of Contents

Search

  1. Preface
  2. Introduction to Data Engineering Streaming
  3. Data Engineering Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Transformation in Streaming Mappings
  8. Window Transformation
  9. Appendix A: Connections
  10. Appendix B: Monitoring REST API Reference
  11. Appendix C: Sample Files

Sources in a Streaming Mapping on Hadoop and Databricks

Sources in a Streaming Mapping on Hadoop and Databricks

A streaming mapping that runs in the Hadoop environment can include file, database, and streaming sources. A streaming mapping that runs in the Databricks environment can include streaming sources.
The following table lists the physical data objects that you can create based on the type of source you read from in either Hadoop or Databricks environment:
Sources in a Streaming Mapping
Environment
Amazon Kinesis
Hadoop, Databricks
Azure Event Hubs
Hadoop, Databricks
Confluent Kafka
Hadoop, Databricks
Google PubSub
Hadoop
HBase
Hadoop
JMS
Hadoop
Kafka
Hadoop, Databricks
MapR Streams
Hadoop
Intelligent structure model is supported for Amazon Kinesis, Azure Event Hubs, Confluent Kafka, and Kafka data objects for streaming mappings that run on the Databricks Spark engine.
You can run streaming mappings in the Databricks environment on the AWS or Azure platforms. The following table shows the list of sources that you can include in a streaming mapping based on the cloud platform:
Sources
Cloud Platform
Amazon Kinesis
AWS
Azure Event Hubs
Azure
Confluent Kafka
AWS, Azure
Kafka
AWS, Azure