In an application, the file rollover process closes the current file and creates a new file on the basis of file size or time. An active file is a file to which the source application writes data. Archived files are the rolled over files.
A file source service tracks the active file and consumes data in real time. If archive files exist when you start the source service, the source service starts reading from the oldest archive file that matches the specified regular expression. The source service then progresses toward the active file in chronological order.
If the chronological order of all files are the same, the file source service processes the files in lexical order. For example, if you name the files log.1, log.2, and log.10, the file source service processes the files in the order log.1, log.10, and log.2.
If you want the file source service to process the files in rollover order and you use the regular expression
log\..*
, you can name the files based on the rollover precision. For example, if the maximum backup index is 1000, you can name the files log.0001, log.0002, and log.0010. Or, if the maximum backup index is 10, you can name the files log.01, log.02, and log.10. In this case, though the source service processes the files in lexical order, it maintains the rollover order.
If a rollover event occurs when the source service that processes the files is down, you might lose data. To avoid data loss, when you configure the source application for log file rollover, verify that the time between two rollover events exceeds maintenance periods.
You can use the File source service in the following rollover scenarios:
Name of the active file does not change after rollover
In this rollover scenario, an application writes data to only one file, for example,
alog.log
. When the rollover occurs, based on the criteria, the application renames the file as
alog.log.1
and continues to write to
alog.log
. In this scenario, configure the File source service to read from
alog.log
file. Specify the following properties for the File source service:
Name. Name for the file source service.
Directory. Path to the directory where the source files are located.
File Name. Name of the active file.
Delimiter. Delimiter that separates entries in the source file.
In this rollover scenario, when you undeploy the data flow or stop the EDS Node mapped to the file source, the application continues to write to the active file
alog.log
. When rollover occurs, the application creates new files. When you redeploy the data flow or start the EDS Node, the source service continues to read from the
alog.log
file and the data in the rolled over files is not read.
When the data flow is deployed and the source service is reading from the
alog.log
file, if the application performs a rapid rollover, then the data in the rolled over files is not read.
Name of the active file changes
In this rollover scenario, an application writes data to a new file and the File source service reads from that file. For example, if the application writes data to a file named
alog.log-<timestamp1>
, after a time based rollover the application writes to a new file named
alog.log-<timestamp2>
. In this scenario, configure the File source service to read from a file that changes according to a predefined pattern. Specify the following properties for the File source service:
Name. Name for the file source service.
Directory. Path to the directory where the source files are located.
File Name. Name of the active file.
File Name is Regular Expression. Java Regular expression that applies to the convention used for naming the active source file.
Delimiter. Delimiter that separates entries in the source file.