PowerExchange for Amazon S3 User Guide

10.5.3
- 10.5.9
- 10.5.8
- 10.5.7
- 10.5.6
- 10.5.5
- 10.5.4
- 10.5.2
- 10.5.1
- 10.5
- 10.4.1
- 10.4.0

Back Next

Hadoop Performance Tuning Options for EMR Distribution

You can use Hadoop Performance Tuning Options to optimize the performance in the Amazon EMR distribution when you copy large volumes of data between Amazon S3 buckets and HDFS.

You must provide semicolon separated name-value attribute pairs for Hadoop Performance Tuning Options.

Use the following parameters for Hadoop Performance Tuning Options:

mapreduce.map.java.opts

fs.s3a.fast.upload

fs.s3a.multipartthreshold

fs.s3a.multipartsize

mapreduce.map.memory.mb

The following sample shows the recommended values for the parameter:

mapreduce.map.java.opts=-Xmx4096m;fs.s3a.fast.upload=true;fs.s3a.multipart.threshold=33554432;fs.s3a.multipart.size=33554432;mapreduce.map.memory.mb=4096

PowerExchange for Amazon S3 Data Objects

Download Guide

Watch

Comments

Communities

Knowledge Base

Success Portal