The Logical Data Transformation Manager translates the mapping into a Scala program, packages it as an application, and sends it to the Spark executor.
The Spark executor submits the application to the Resource Manager in the Hadoop cluster and requests resources to run the application.
When you run mappings on the HDInsight cluster, the Spark executor launches a spark-submit script. The script requests resources to run the application.
The Resource Manager identifies the Node Managers that can provide resources, and it assigns jobs to the data nodes.
Driver and Executor processes are launched in data nodes where the Spark application runs.