Lookup Transformation on the Databricks Spark Engine
Lookup Transformation on the Databricks Spark Engine
Some processing rules for the Databricks Spark engine differ from the processing rules for the Data Integration Service.
Multiple Matches
When you choose to return the first, last, or any value on multiple matches, the Lookup transformation returns any value.
If you configure the transformation to report an error on multiple matches, the Databricks Spark engine drops the duplicate rows and does not include the rows in the logs.
If an HBase lookup does not result in a match, it generates a row with null values for all columns. You can add a Filter transformation after the Lookup transformation to filter out null rows.
Mapping Validation
Mapping validation fails in the following situations:
Case sensitivity is disabled.
The lookup condition contains binary data type.
The cache is configured to be shared, named, persistent, dynamic, or uncached. The cache must be a static cache.
The lookup condition uses a field with a complex data type.
The lookup source is not Microsoft Azure SQL Data Warehouse.
If an HBase lookup does not result in a match, it generates a row with null values for all columns. You can add a Filter transformation after the Lookup transformation to filter out null rows.