Rules and guidelines for personal staging location
Rules and guidelines for personal staging location
When you select the personal staging location as a staging environment, the data is first staged in a java temporary location and then copied to a personal staging location of the unity catalog. Both the staged files will be deleted after the
task
runs successfully.
However, to stage the data in a different directory, configure the DTM property
-Djava.io.tmpdir=/my/dir/path
in the JVM options in the system configuration settings of the Administrator service.
To enable data staging in a different directory, you should have read and write permission and enough disk space to stage the data in the directory.
When you specify a personal staging location in the Databricks Delta connection properties for staging, consider the following rules and guidelines:
You can only specify unity enabled catalog in the SQL warehouse JDBC URL.
All mappings that are configured run without
SQL ELT optimization
.
The data is staged in the folder
stage://tmp/<user_name>
where the
<user_name>
is picked from the Databricks token provided in the connection and requires read and write access to the personal staging location in root location of AWS and Azure.