Target File Rollover
When you write to an HDFS target, you can perform the following types of rollover:
- Size-based rollover
- You can configure an HDFS target service to perform target file rollover when the target file reaches a certain size. To configure size-based rollover, specify the
Rollover Size property of the target service.
- Time-based rollover
- You can configure an HDFS target service to perform target file rollover when a certain period of time has elapsed after the target service creates the target file. To configure time-based rollover, specify the
Rollover Time property of the target service.
You can implement both rollover schemes for a target file, in which case, the event that occurs first triggers a rollover. For example, if you set rollover time to 1 hour and rollover size to 1 GB, the target service rolls the file over when the file reaches a size of 1 GB even if the 1-hour period has not elapsed.
The target file to which the target service is writing data is the active file. Files that the target service closes after performing a rollover are archived files. When the target service archives a file, it appends the timestamp to the file name. For example, set the rollover size as 1 MB and name of the file target as
target.log. If the source service sends 5 MB to the file target, the file target first creates the
target.log.<timestamp> file. When the size of
target.log.<timestamp> reaches 1 MB, the target service rolls the file over.