Advanced Clusters

Back Next

Running a job without direct access to Amazon data sources

To run a job that doesn't use a connector with direct access to Amazon data sources, the cluster accesses Amazon resources using the connection properties and the worker role.

For example, JDBC V2 Connector doesn't have direct access to Amazon data sources. To run a job that uses JDBC V2 Connector, the cluster uses the connection properties to read and temporarily stage the data before processing and writing the data to the target.

The following image shows the process that the Secure Agent and cluster nodes use to run the job:

The following steps describe the process that the Secure Agent and cluster nodes use to run the job:

The Secure Agent assumes the cluster operator role to store job dependencies in the staging location.

The worker nodes use connection properties to access source data.

The worker nodes use the worker role to access the staging location to get job dependencies and stage temporary data.

The worker nodes use the worker role to auto-scale EBS volumes if the job requires more storage space.

The master node uses the master role to scale cluster nodes based on resource requirements.

The worker nodes use the worker role to store logs in the log location.

The master node uses the master role to store logs in the log location.

The Secure Agent uses the Secure Agent role to upload the agent job log to the log location.

If any connector in the job uses AWS credentials to directly access a source or target, the connection-level AWS credentials override the worker role to gain access to the staging location.

Rename Saved Search

Table of Contents

Advanced Clusters

Advanced Clusters

Running a job without direct access to Amazon data sources

Running a job without direct access to Amazon data sources