Table of Contents

Search

  1. Preface
  2. Advanced clusters
  3. Setting up AWS
  4. Setting up Google Cloud
  5. Setting up Microsoft Azure
  6. Setting up a self-service cluster
  7. Setting up a local cluster
  8. Advanced configurations
  9. Troubleshooting
  10. Appendix A: Command reference

Advanced Clusters

Advanced Clusters

Default staging and log locations

Default staging and log locations

A default local cluster stores staging and log files on the agent machine when you run jobs on the cluster.
The following table lists the default staging and log locations:
Default location
Description
file:///$ADVANCED_MODE_STAGING
Default staging location in the following directory on the agent machine:
<agent installation directory>/apps/Advanced_Mode_Staging_Dir
file:///$ADVANCED_MODE_LOG
Default log location in the following directory on the agent machine:
<agent installation directory>/apps/Advanced_Mode_Log_Dir
The agent machine must have enough space in the default staging and log locations so that jobs can run successfully. You can change the staging and log locations by editing the
advanced configuration
for the cluster.
If
Data Integration
creates a subtask for the Data Integration Server to process data logic in a mapping in advanced mode, the cluster and the Data Integration Server share staging files. To read and write staging files in the staging location, the Data Integration Server uses Hadoop Files V2 Connector.

0 COMMENTS

We’d like to hear from you!