Databricks Connector

Back Next

Mappings for Databricks

When you configure a mapping, you describe the flow of data from the source to the target.

A mapping defines reusable data flow logic that you can use in mapping tasks.

When you create a mapping, you define the Source, Target, and Lookup transformations to represent a

Databricks

object. Use the Mapping Designer in Data Integration to add the Source, Target, or Lookup transformations in the mapping canvas and configure the

Databricks

source, target, and lookup properties.

After you create a mapping, you can run the mapping or you can deploy the mapping in a

mapping

task. The

mapping

task allows you to process data based on the data flow logic defined in a mapping.

In advanced mode, the Mapping Designer updates the mapping canvas to include transformations and functions that enable advanced functionality.

You can use Monitor to monitor the jobs.

The following table lists the functionalities that are supported by SQL warehouse, all-purpose cluster, and job cluster:

Property	SQL warehouse¹	All-purpose cluster²	Job cluster³
Source transformation	Yes	Yes	Yes
Target transformation	Yes	Yes	Yes
Filter transformation	Yes	No	Yes
Lookup transformation	Yes	No	No
Sorter transformation	Yes	No	Yes
SQL transformation	Yes	No	No
Dynamic schema handling	Yes	No	No
Identity columns	Yes	No	No
¹The Secure Agent connects to the SQL warehouse at design time and runtime. ²The Secure Agent connects to the all-purpose cluster to import the metadata at design time. ³The Secure Agent connects to the job cluster to run the mappings.