Data Engineering Streaming
- Data Engineering Streaming 10.4.1
- All Products
Indicates the type of data object operation.
This is a read-only property.
The location of the complex file target.
At run time, the Data Integration Service creates temporary directories in the specified file directory to manage the target files.
If the directory is in HDFS, enter the path without the node URI. For example,
/user/lib/testdirspecifies the location of a directory in HDFS. The path must be 512 characters or less.
Not applicable for streaming mappings.
The name of the output file. Spark appends the file name with a unique identifier before it writes the file to HDFS.
The file format. Select one of the following file formats:
The class name for files of the output format. If you select Output Format in the
File Formatfield, you must specify the fully qualified class name implementing the
Output Key Class
The class name for the output key. By default, the output key class is NullWritable.
Output Value Class
The class name for the output value. By default, the output value class is Text.
Optional. The compression format for binary files. Select one of the following options:
Custom Compression Codec
Required for custom compression. Specify the fully qualified class name implementing the
Sequence File Compression Type
Optional. The compression format for sequence files. Select one of the following options:
Stream Rollover Size in GB
Optional. Target file size, in gigabytes (GB), at which to trigger rollover. A value of zero (0) means that the target file does not roll over based on size. Default is 1 GB.
Stream Rollover Time in Hours
Optional. Length of time, in hours, for a target file to roll over. After the time period has elapsed, the target file rolls over. A value of zero (0) means that the target file does not roll over based on time. Default is 1 Hour.
Optional. The schema location to fetch the schema in a streaming mapping.
Only Avro schema using binary file format is supported. You must disable the column projection.
If you select
External Locationfor the dynamic schema strategy, you must create a
writer.avscfile having the schema content at the schema location and keep it under the topic name. For example:
<Schema Location>/<Topic Name>/writer.avsc. Then, specify only the path till the schema location.
Schema location must be named as per the topic name.
Dynamic Schema Strategyis enabled at source and schema location is not provided for the HDFS target, then at runtime schema location fetches the schema for the HDFS target from the source transformation schema.
The active directory location of the complex file target. This directory stages all the files currently in open state. When the stream rollover condition is met, the files are moved from the interim directory to the target directory.
Updated March 14, 2021