The study uses the following cluster workflow:

The cluster workflow contains three mapping tasks,

Task 1

,

Task 2

, and

Task 3

. The mapping tasks are identical mappings that process the same data volume. The tasks are executed sequentially.

The workflow is submitted to an ephemeral Amazon EMR cluster that employs auto-scaling. The auto-scaling rules have a minimum capacity set to 4 nodes and a maximum capacity set to 10 nodes. The auto-scaling rules add nodes to the cluster in increments of 2.

The following image shows the auto-scaling policy:

The following performance chart shows a timeline of the workflow execution on the Amazon EMR cluster:

The performance chart shows the following events:

Creating an ephemeral cluster took 7 minutes.
The first mapping task ran on 4 data nodes and took around 13 minutes to complete.
While the first task ran, the auto-scaling policy was prompted to add 2 data nodes to the cluster, but it took around 8 minutes for the additionally commissioned nodes to be available for processing.
By the time that the additional nodes became available, the execution of the first mapping task was completed.

The second mapping task leveraged the additional nodes and ran on 6 data nodes. The additional computational resources reduced the execution run time from around 13 minutes to around 8 minutes.

Auto-scaling rules for ephemeral clusters on AWS can be defined in multiples of 5 minutes. Once the auto-scaling rules are prompted, it can take between 5 to 10 minutes for additionally commissioned nodes to become available.

Based on these observations, Informatica recommends that you implement auto-scaling rules only when processing large volumes of data, such as Spark applications that run mapping jobs over multiple iterations.