Rules and Guidelines for Mappings on the Databricks Spark Engine
Rules and Guidelines for Mappings on the Databricks Spark Engine
Consider the following run-time differences on the Databricks Spark engine:
Set the optimizer level to none or minimal if a mapping validates but fails to run. If you set the optimizer level to use cost-based or semi-join optimization methods, the Data Integration Service ignores this at run-time and uses the default.
The run-time engine does not honor the early projection optimization method in all cases. If the Data Integration Service removes the links between unused ports, the run-time engine might reconnect the ports.
When you use the auto optimizer level, the early selection optimization method is enabled if the mapping contains any data source that supports pushing filters to the source on the Spark or Databricks Spark engines. For more information about optimizer levels, see the