What's New and Changed (10.5.4.0.1)

Back Next

Streaming Mappings in Azure Databricks

Effective in version 10.4.0, you can run streaming mappings in Azure Databricks service in Microsoft Azure cloud services.

Sources and Targets: You can run streaming mappings against the following sources and targets within the Databricks environment:

Microsoft Azure Event Hubs
Azure Data Lake Storage Gen2 (ADLS Gen2)
Transformations: You can add the following transformations to a Databricks streaming mapping:

Aggregator
Expression
Filter
Joiner
Normalizer
Rank
Router
Union
Window

Data Types: The following data types are supported:
Array
Bigint
Date/time
Decimal
Double
Integer
Map
Struct
Text
String

Workflows: You can develop cluster workflows to create ephemeral clusters in the Databricks environment. Use Azure Data Lake Storage Gen1 (ADLS Gen1) and Azure Data Lake Storage Gen2 (ADLS Gen2) to create ephemeral clusters in the Databricks environment.

For more information about streaming mappings in Azure Databricks, see the

Data Engineering Streaming 10.4.0 User Guide

Watch

Comments