To add a file source service, use the File source service type.
You can configure the following properties for the File source service type:
- Entity Name
- Name of the source service. Maximum length is 32 characters.
- Description of the source service. Maximum length is 256 characters.
- Full path to the directory that contains the source files.
- Position File Directory
- Full path to the directory where EDS writes the position file. Specify this directory if the source file directory does not have write permissions.
- File Name
- Name of the source file or the Java regular expression that applies to the convention used to name the active source file.
- File Name is Regular Expression
- Indicates that the file name is a Java regular expression.
- Processed File Directory
- Optional. Location where the files should be moved after they are processed. Specify the location if the file name is a Java regular expression.
If you do not specify a location, the files will not be moved.
- A character or sequence of characters that separates the data.
You can choose one of the following delimiters:
- LF. Line feed character sequence.
- CRLF. Carriage return line feed character sequence.
- Custom. Custom character sequence.
Default is LF.
- Custom Delimiter
- Custom character sequence that you want to specify as delimiter for the file source services.
Enter a custom character sequence if you select
Custom as the delimiter.
- Maximum Event Size
- Maximum length of data that the source service can read at a time, in bytes. Default is 8192. Minimum value is 1. Maximum value is 51200.
- Retry on Failure
- Indicates that the target service should try to write to the target if the operation fails the first time.
- Number of Retries
- The number of times the source service should retry to open or read from the source.
- Delay between Retries
- Time in milliseconds between successive retries.
- Persist Data
- Persist the data from the source locally so that the source service can continue to process data even if the target is not available.
- Persistence Options
- You can configure the following persistence options:
- batchSizeInBytes. The batch size of the events in bytes after which the data is written to the persistent store. Specify a value of 0 if you want to write the data to the persistent store immediately. Default is 262144 bytes.
- flushInterval. The interval in milliseconds after which the source services writes data to the persistent store. Default is 5000 milliseconds.
- maxDataFiles. The maximum number of data files that the source service can keep in the persistent store. After the number of files exceed this number, a rollover occurs. Default is 10.
- maxDataFileSize. The maximum size of each data file in bytes. When the file reaches the specified size, a file rollover occurs. Default is 1073741824.
- maxQueueSizeInBytes. The maximum size of data that is unsent or has not received acknowledgment in bytes that can be stored in the persistent store. If the specified size is exceeded, the source service does not read data from the source. If you specify a value of 0, the unsent data can be as much as the value of
maxDataFileSize. Default is unlimited size.
- You can monitor the following statistics:
- Bytes Sent. The number of bytes sent by the source service.
- Events Sent. The number of events sent by the source service.
- Events to be Sent. The number of events that the source service is yet to send.
- Events not Delivered. Number of events that the source service did not deliver.
- Send Rate (Per Sec.). The number of bytes sent every second.
- Events Dropped. The number of events dropped while processing the source data.
- Files Written to Target. The number of files sent to the target service by the source service.
- Files to be Processed. The number of files to be processed. This number includes the file that is being processed and the pending files.
- Error while Moving Files. The number of errors that occur while moving the files.