Table of Contents

Search

  1. Preface
  2. Part 1: Version 10.4.1
  3. Part 2: Version 10.4.0
  4. Part 3: Version 10.2.2
  5. Part 4: Version 10.2.1
  6. Part 5: Version 10.2
  7. Part 6: Version 10.1.1
  8. Part 7: Version 10.1

Streaming Mappings in Azure Databricks

Streaming Mappings in Azure Databricks

Effective in version 10.4.0, you can run streaming mappings in Azure Databricks service in Microsoft Azure cloud services.
Sources and Targets
You can run streaming mappings against the following sources and targets within the Databricks environment:

    Microsoft Azure Event Hubs

    Azure Data Lake Storage Gen2 (ADLS Gen2)

Transformations
You can add the following transformations to a Databricks streaming mapping:

    Aggregator

    Expression

    Filter

    Joiner

    Normalizer

    Rank

    Router

    Union

    Window

Data Types
The following data types are supported:

    Array

    Bigint

    Date/time

    Decimal

    Double

    Integer

    Map

    Struct

    Text

    String

Workflows
You can develop cluster workflows to create ephemeral clusters in the Databricks environment. Use Azure Data Lake Storage Gen1 (ADLS Gen1) and Azure Data Lake Storage Gen2 (ADLS Gen2) to create ephemeral clusters in the Databricks environment.
For more information about streaming mappings in Azure Databricks, see the
Data Engineering Streaming 10.4.0 User Guide
.


Updated August 28, 2020