REST API Reference

Back Next

Connection configuration for tasks with flat file as a source and as a target

When the source and target connection of the task is flat file, you can include the following fields and key-value pairs in the config array of the source and target connection:

Key	Type	Required	Description
File	String	Yes	Absolute path and name of the source file. Enter the base directory for multiple files mode.
initialPosition	String	Yes	Starting position to read data from the file to tail. Enter one of the following values: Beginning of File. Read from the beginning of the file. Don't ingest any data that has already been rolled over. Current Time. Read from the most recently updated part of the file. Don't ingest data that was rolled over or data in the file that was written.
rolloverPattern	String	-	File name pattern for the file that rolls over. If the file to tail rolls over, the Secure Agent uses the file name pattern to identify files that have rolled over. If the Secure Agent stops during a file rollover, when it restarts, it picks up the file where it was left off. You can use asterisk () and question mark (?) as wildcard characters to indicate that the files are rolled over in the same directory. For example, ${filename}.log. . Here, asterisk (*) represents the successive version numbers that would be appended to the file name.
tailingMode	String	Yes	Tail a file or multiple files based on the logging pattern. Enter one of the following values: Single file. Tail one file. Multiple files. Tail all the files indicated in the base directory. You can enter a regular expression to indicate the files to tail.
File Name	String	Yes	The name of the target file.
interimDirectory	String	Yes	Path to the staging directory on the Secure Agent.
rolloverSize	Integer	Yes	The file size, in KB, at which the task moves the file from the staging directory to the target. You can enter a value between 1 and 2147483647.
rolloverEvents	Integer	Yes	Number of events or messages to accumulate before a file rollover. You can enter a value between 1 and 2147483647.
rolloverTime	Integer	-	Length of time, in milliseconds, after which the target file rolls over. You can enter a value between 1 and 2147483647.
edges	Array	-	Sequence of dataflow execution.

POST request example

To update a

streaming ingestion and replication

task with a flat file source and target, you might send a request similar to the following example:

{
  "name": "FileToFile",
  "description": "FileToFile_V2",
  "runtimeId": "01000025000000000003",
  "locationId": "5sJ0JDyJyWLlrosS5qJjsQ",
  "currentVersion": "2",
  "messageFormat": "binary",
  "nodes": [
    {
      "name": "FileToFile_source",
      "type": "source",
      "connectionId": "0100000B000000000002",
      "transformationType": "",
      "config": [
        {
          "key": "File",
          "value": "siagent.log"
        },
        {
          "key": "initialPosition",
          "value": "Current Time"
        },
        {
          "key": "rolloverPattern",
          "value": ""
        },
        {
          "key": "tailingMode",
          "value": "Single file"
        }
      ]
    },
    {
      "name": "FileToFile_target",
      "type": "target",
      "connectionId": "0100000B000000000002",
      "transformationType": "",
      "config": [
        {
          "key": "File Name",
          "value": "testing.log"
        },
        {
          "key": "interimDirectory",
          "value": "/home/agent/infa/test_file_target"
        },
        {
          "key": "rolloverSize",
          "value": 100
        },
        {
          "key": "rolloverEvents",
          "value": 100
        },
        {
          "key": "rolloverTime",
          "value": 100
        }
      ]
    }
  ],
  "edges": [
    {
      "from": "FileToFile_source",
      "to": "FileToFile_target"
    }
  ],
  "runtimeOptions": {
    "maxLogSize": {
      "value": 10,
      "unit": "MB"
    },
    "logLevel": "INFO"
  }
}

Rename Saved Search

Table of Contents

REST API Reference

REST API Reference

Connection configuration for tasks with flat file as a source and as a target

Connection configuration for tasks with flat file as a source and as a target

POST request example