Table of Contents

Search

  1. Preface
  2. Components
  3. API collections
  4. Business services
  5. File listeners
  6. Fixed-width file formats
  7. Hierarchical schemas
  8. Intelligent structure models
  9. Refining intelligent structure models
  10. Mapplets
  11. Saved queries
  12. Shared sequences
  13. User-defined functions

Components

Components

File event reliability

File event reliability

When you use a file listener as a source in a file ingestion task, it creates a file event when new files arrive in the folder that it monitors and when files in the folder are updated or deleted. The file listener notifies the file ingestion task of the new or updated files.
The file listener handles the events based on the following conditions:
  • If multiple events occur, the file listener notifies the file ingestion task with only the last event for each file.
  • If the Secure Agent isn't running or there is a temporary network disruption, and file events don't reach the file ingestion task, the file listener queues the last event for each file and includes it in the notification of the next file ingestion job. A file ingestion task thus receives a notification about each file at least once. This ensures at-least-once reliability between the file listener and the file ingestion task.
    File events that aren't processed remain in the queue for seven days.
  • File events that are in the file listener queue reach the file ingestion task by one of the following methods:
    • When a file ingestion job completes, the mass ingestion service makes a pull request to the file listener to check for any queued events. If it finds any events, the service triggers a new ingestion job to process them. The pull request doesn't trigger the processing of files that are already assigned to another concurrent job that runs by the same ingestion task, so only one ingestion job processes a file at any time.
    • If any events aren't picked up by the proactive pull request, for example, if the Secure Agent isn't running when the mass ingestion service makes the request, the file listener queues the last event for each file and includes it in the notification of the next file ingestion job.
  • The file ingestion task doesn't automatically reprocess file events that are in success, failed, or duplicate status. If an event in one of these statuses reaches the file ingestion task, the file ingestion job informs the file listener that the event is processed even if it encountered errors in processing the file or in writing it to the target. Therefore, at-least-once reliability exists only between the file listener and the file ingestion task, and not between the file listener and the target.
    You need to manually identify files that aren't successfully transferred to the target due to an error, for example, by using the transfer logs. To resolve the problem, either move the files or modify them manually, so that the file listener picks them up. For example, if the last modified time of a file changes, the file listener identifies the file as updated even if the contents haven't changed.

0 COMMENTS

We’d like to hear from you!