PowerExchange for HDFS User Guide

10.5.6
- 10.5.9
- 10.5.8
- 10.5.7
- 10.5.4
- 10.5.3
- 10.5.1
- 10.5
- 10.4.1
- 10.4.0

Back Next

Compression and Decompression for Flat File Sources and Targets

File compression can increase data transfer rates and reduce space for data storage.

You can read and write compressed flat files, specify compression formats, and decompress files. You can compress and decompress files in compression formats such as Bzip2 and Lzo, or specify a custom compression format.

You can specify a file or a directory of files. When the Data Integration Service reads from a directory, it reads the files of the specified format only and ignores files of other formats.

For information about how Hadoop processes compressed and uncompressed files, see the Hadoop documentation.

The following table describes the compression options:

Compression Options	Description
None	The file is not compressed.
Auto	The Data Integration Service detects the compression format of the file based on the file extension.
Gzip	The GNU zip compression format that uses the DEFLATE algorithm.
Bzip2	The Bzip2 compression format that uses the Burrows–Wheeler algorithm.
Lzo	The Lzo compression format that uses the Lempel-Ziv-Oberhumer algorithm. The JAR files for LZO compression are not available with the default Hadoop installation. You must place the JAR files for the LZO compression format in the lib folder of the distribution directory and verify the distribution directory properties.
Custom	Custom compression format. If you select this option, you must specify the fully qualified class name implementing the Hadoop CompressionCodec interface in the Compression Codec field.

Rename Saved Search

Table of Contents

PowerExchange for HDFS User Guide

PowerExchange for HDFS User Guide

Compression and Decompression for Flat File Sources and Targets

Compression and Decompression for Flat File Sources and Targets