Rules and Guidelines for Mappings on the Databricks Spark Engine
Rules and Guidelines for Mappings on the Databricks Spark Engine
A non-native environment like Databricks Spark uses distributed processing and processes data on different nodes. Each node does not have access to the data that is being processed on other nodes. As a result, the runtime engine might not be able to determine the order in which the data originated. So, when you run a mapping in a non-native environment and you run the same mapping in the native environment, both mappings return correct results, but the results might not be identical. For more information, see the
Consider the following run-time differences on the Databricks Spark engine:
Spark engine
Set the optimizer level to none or minimal if a mapping validates but fails to run. If you set the optimizer level to use cost-based or semi-join optimization methods, the Data Integration Service ignores this at run-time and uses the default.
The run-time engine does not honor the early projection optimization method in all cases. If the Data Integration Service removes the links between unused ports, the run-time engine might reconnect the ports.
Databricks Spark performs auto-optimization on jobs based on configuration settings for the cluster. If you set optimization through the Spark.default.parallelism property, the Databricks Spark engine ignores this setting. As a consequence, it is not possible to configure optimization on the job level.
When you use the auto optimizer level, the early selection optimization method is enabled if the mapping contains any data source that supports pushing filters to the source on Spark or Databricks Spark engines. For more information about optimizer levels, see the
The run-time engine drops rejected rows even if you configure the mapping to forward rejected rows. The rejected rows are not written to the session log file.
When the Spark engine runs a mapping, it processes jobs on the cluster using HiveServer2 in the following cases:
The mapping writes to a target that is a Hive table bucketed on fields of type char or varchar.
The mapping reads from or writes to Hive transaction-enabled tables.
The mapping reads from or writes to Hive tables where column-level security is enabled.
The mapping writes to a Hive target and is configured to create or replace the table at run time.