Rules and Guidelines for Mappings on the Databricks Spark Engine
Rules and Guidelines for Mappings on the Databricks Spark Engine
Consider the following run-time differences on the Databricks Spark engine:
Set the optimizer level to none or minimal if a mapping validates but fails to run. If you set the optimizer level to use cost-based or semi-join optimization methods, the Data Integration Service ignores this at run-time and uses the default.
The run-time engine does not honor the early projection optimization method in all cases. If the Data Integration Service removes the links between unused ports, the run-time engine might reconnect the ports.