Table of Contents

Search

  1. Preface
  2. Introduction to Amazon S3 V2 Connector
  3. Connections for Amazon S3 V2
  4. Amazon S3 V2 sources and targets
  5. Mappings and mapping tasks with Amazon S3 V2
  6. Migrating a mapping
  7. Upgrading to Amazon S3 V2 Connector
  8. Data type reference
  9. Troubleshooting

Amazon S3 V2 Connector

Amazon S3 V2 Connector

Data compression in Amazon S3 V2 sources and targets

Data compression in Amazon S3 V2 sources and targets

You can decompress data when you read data from Amazon S3
and compress the data when you write data to Amazon S3
.
The following table lists the supported source compression formats:
Compression format
Avro File
Binary File
1
Flat
JSON File
2
ORC File
Parquet File
None
Yes
No
Yes
Yes
Yes
Yes
Bzip2
No
No
No
Yes
No
No
Deflate
Yes
No
No
No
No
No
Gzip
No
No
Yes
No
No
Yes
Lzo
No
No
No
No
No
No
Snappy
Yes
No
No
No
Yes
Yes
Zlib
No
No
No
No
Yes
No
Binary File support doesn't apply to mappings in advanced mode, while the JSON File support applies only to mappings in advanced mode.
The following table lists the supported target compression formats:
Compression format
Avro File
Binary File
1
Flat
JSON File
2
ORC File
Parquet File
None
Yes
No
Yes
Yes
Yes
Yes
Bzip2
No
No
No
Yes
No
No
Deflate
Yes
No
No
Yes
No
No
Gzip
No
No
Yes
Yes
No
Yes
Lzo
No
No
No
No
No
No
Snappy
Yes
No
No
Yes
Yes
Yes
Zlib
No
No
No
No
Yes
No
1
Doesn't apply to mappings in advanced mode.
2
Applies only to mappings in advanced mode.
Configure the compression format in the
Compression Format
option under the advanced source
and target
properties.
For the Avro, ORC and Parquet file formats, the support for the following compression formats are implicit even though these compression formats do not appear in the
Compression Format
option under the advanced source property:
Compression format
Avro File
ORC File
Parquet File
Deflate
Yes
No
No
Snappy
Yes
Yes
Yes
Zlib
No
Yes
No

0 COMMENTS

We’d like to hear from you!