Amazon S3 V2 Connector

Back Next

Data compression in Amazon S3 V2 sources and targets

You can decompress data when you read data from Amazon S3

and compress the data when you write data to Amazon S3

Configure the compression format in the

Compression Format

option under the advanced source

and target

properties.

Compression format doesn't apply to binary files.

The following table lists the compression formats that you can use for various file types when you read data from Amazon S3:

Compression format	File type
None	Avro, Delta¹, Flat, JSON², ORC, Parquet
Bzip2	JSON²
Deflate	Avro
Gzip	Delta¹, Flat, Parquet
Lzo	Doesn't apply to any file type.
Snappy	Avro, Delta¹, ORC, Parquet
Zlib	ORC
¹Doesn't apply to mappings in advanced mode. ²Applies only to mappings in advanced mode.

The following table lists the compression formats that you can use for various file types when you write data to Amazon S3:

Compression format	File type
None	Avro, Delta¹, Flat, JSON², ORC, Parquet
Bzip2	JSON²
Deflate	Avro, JSON²
Gzip	Delta¹, Flat, JSON², Parquet
Lzo	Doesn't apply to any file type.
Snappy	Avro, Delta¹, JSON², ORC, Parquet
Zlib	ORC
¹Doesn't apply to mappings in advanced mode. ²Applies only to mappings in advanced mode.

For the Avro, ORC and Parquet file formats, the support for the following compression formats are implicit even though these compression formats do not appear in the

Compression Format

option under the advanced source property: