Table of Contents

Search

  1. Preface
  2. Introduction to Big Data Streaming
  3. Big Data Streaming Configuration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Appendix A: Connections
  9. Appendix B: Data Type Reference
  10. Appendix C: Sample Files

Big Data Streaming User Guide

Big Data Streaming User Guide

Advanced Properties

Advanced Properties

The Developer tool displays the advanced properties for complex file targets in the Input transformation in the
Write
view.
The following table describes the advanced properties that you configure for complex file targets in a streaming mapping:
Property
Description
Operation Type
Indicates the type of data object operation.
This is a read-only property.
File Directory
The directory location of the complex file target.
If the directory is in HDFS, enter the path without the node URI. For example,
/user/lib/testdir
specifies the location of a directory in HDFS. The path must not contain more than 512 characters.
If the directory is in the local system, enter the fully qualified path. For example,
/user/testdir
specifies the location of a directory in the local system.
The Data Integration Service ignores any subdirectories and their contents.
File Name
The name of the output file. Spark appends the file name with a unique identifier before it writes the file to HDFS.
File Format
The file format. Select one of the following file formats:
  • Binary. Select Binary to read any file format.
  • Sequence. Select Sequence File Format for target files of a specific format that contain key and value pairs.
Output Format
The class name for files of the output format. If you select Output Format in the
File Format
field, you must specify the fully qualified class name implementing the
OutputFormat
interface.
Output Key Class
The class name for the output key. By default, the output key class is NullWritable.
Output Value Class
The class name for the output value. By default, the output value class is Text.
Compression Format
Optional. The compression format for binary files. Select one of the following options:
  • None
  • Auto
  • DEFLATE
  • gzip
  • bzip2
  • LZO
  • Snappy
  • Custom
Custom Compression Codec
Required for custom compression. Specify the fully qualified class name implementing the
CompressionCodec
interface.
Sequence File Compression Type
Optional. The compression format for sequence files. Select one of the following options:
  • None
  • Record
  • Block