Table of Contents

Search

  1. Preface
  2. Advanced clusters
  3. Setting up AWS
  4. Setting up Google Cloud
  5. Setting up Microsoft Azure
  6. Setting up a self-service cluster
  7. Setting up a local cluster
  8. Advanced configurations
  9. Troubleshooting
  10. Appendix A: Command reference

Advanced Clusters

Advanced Clusters

Local cluster properties

Local cluster properties

Create an
advanced configuration
to configure properties for an
advanced cluster
. The properties describe where you want to start the cluster on your cloud platform and the infrastructure that you want to use.
The basic properties describe the
advanced configuration
. To configure the cluster, configure the platform and runtime properties.

Basic configuration

The following table describes the basic properties:
Property
Description
Name
Name of the
advanced configuration
.
Description
Description of the
advanced configuration
.
Runtime Environment
Runtime environment to associate with the
advanced configuration
. The runtime environment can contain only one Secure Agent. A runtime environment cannot be associated with more than one configuration.
Cloud Platform
Cloud platform that hosts the cluster.
Select Local.

Platform configuration

The following table describes the platform properties:
Property
Description
Mapping
Task Timeout
Amount of time to wait for a
mapping
task to complete before it is terminated. By default, a
mapping
task does not have a timeout.
If you specify a timeout, a value of at least 10 minutes is recommended. The timeout begins when the
mapping
task is submitted to the Secure Agent.
Staging Location
Location of the staging data.
For staging data on a local file system, specify the location in the following format:
file://<absolute path to the Secure Agent location>
For example, to use /home/devbld/staging as the staging location, enter:
file:///home/devbld/staging
Data Integration creates the directory if it does not already exist. Note the extra '/' character from the absolute path.
For staging locations on the cloud, specify the path in one of the following formats:
  • Amazon S3.
    s3://<bucket name>/<folder path>
  • Google Cloud Storage.
    gs://<bucket name>/<folder path>&:<project ID>/<region>
  • Microsoft Azure Data Lake Storage Gen2.
    abfs(s)://<file system>@<storage account>.dfs.core.windows.net/<folder path>&:<resource group>/<region>
The region is optional. For a list of valid regions, refer to your cloud provider's documentation.
The following examples show how region formats might differ on each cloud platform:
  • On AWS, use
    us-west-2
    for US West (Oregon).
  • On Google Cloud, use
    us-west2
    for Los Angeles.
  • On Microsoft Azure, use
    westus2
    for West US 2.
When the Secure Agent creates a local cluster on Oracle Cloud Infrastructure, the staging location must be on the local file system.
Log Location
Location of the logs.
For logs on the local file system, specify the location in the following format:
file://<absolute path to the Secure Agent location>
For example, to use /home/devbld/logging as the log location, enter:
file:///home/devbld/logging
Data Integration creates the directory if it does not already exist. Note the extra '/' character from the absolute path.
For log locations on the cloud, specify the path in the following formats:
  • Amazon S3.
    s3://<bucket name>/<folder path>
  • Google Cloud Storage.
    gs://<bucket name>/<folder path>&:<project ID>/<region>
  • Microsoft Azure Data Lake Storage Gen2.
    abfs(s)://<file system>@<storage account>.dfs.core.windows.net/<folder path>&:<resource group>/<region>
The region is optional. For a list of valid regions, refer to your cloud provider's documentation.
The following examples show how region formats might differ on each cloud platform:
  • On AWS, use
    us-west-2
    for US West (Oregon).
  • On Google Cloud, use
    us-west2
    for Los Angeles.
  • On Microsoft Azure, use
    westus2
    for West US 2.
When the Secure Agent creates a local cluster on Oracle Cloud Infrastructure, the log location must be on the local file system.

Runtime configuration

The following table describes the runtime properties:
Property
Description
Encrypt Data
Indicates whether temporary data on the cluster is encrypted.
Runtime Properties
Custom properties to customize the cluster and the jobs that run on the cluster.

0 COMMENTS

We’d like to hear from you!