Table of Contents

Search

  1. Preface
  2. Advanced clusters
  3. Setting up AWS
  4. Setting up Google Cloud
  5. Setting up Microsoft Azure
  6. Setting up a self-service cluster
  7. Setting up a local cluster
  8. Advanced configurations
  9. Troubleshooting
  10. Appendix A: Command reference

Advanced Clusters

Advanced Clusters

Change staging and log locations (optional)

Change staging and log locations (optional)

When you run jobs on the local cluster, you can choose staging and logging directories on the Secure Agent machine's local file system or on a cloud location. By default, the local cluster uses a local file system path unless you've configured a cloud destination.
To change the staging or log location to a cloud location, complete the following tasks:
  1. Refer to the following table to create the location in your cloud environment:
    Cloud environment
    Create location
    AWS
    Create the following Amazon S3 locations:
    • An S3 location that the cluster uses to store staging files at run time
    • An S3 location that the cluster uses to store log files for the
      advanced jobs
      that run on the cluster
    Microsoft Azure
    Create a storage account using Azure Data Lake Storage Gen2 with locations for staging and log files. Use a hierarchical namespace.
    Google Cloud
    In a Google Cloud environment, create locations for staging and log files on Google Cloud Storage.
  2. Specify the location in the advanced configuration of the advanced cluster. For more information about the format of the staging and log locations, see Local cluster properties

0 COMMENTS

We’d like to hear from you!