Lookup Transformation on the Databricks Spark Engine
Lookup Transformation on the Databricks Spark Engine
Some processing rules for the Databricks Spark engine differ from the processing rules for the Data Integration Service.
Mapping Validation
Mapping validation fails in the following situations:
Case sensitivity is disabled.
The lookup condition in the Lookup transformation contains binary data type.
The cache is configured to be shared, named, persistent, dynamic, or uncached. The cache must be a static cache.
The lookup source is not Microsoft Azure SQL Data Warehouse.
The mapping fails in the following situation:
The transformation is unconnected and used with a Joiner transformation.
Multiple Matches
When you choose to return the first, last, or any value on multiple matches, the Lookup transformation returns any value.
If you configure the transformation to report an error on multiple matches, the Spark engine drops the duplicate rows and does not include the rows in the logs.