REST API Reference

Back Next

Connection configuration for tasks with Microsoft Azure Data Lake Storage Gen2 (ADLS Gen2) as a target

When the target connection of the task target is ADLS Gen2, you can include the following fields and key-value pairs in the config array of the target connection:

Key	Type	Required	Description
writeStrategy	String	Yes	The action to take when a file by the same name exists in the ADLS Gen2 storage. Enter one of the following values: Append. Add data to the existing file. Overwrite. Replaces the existing file with the new file. Fail. Fail the request. Rollover. Close the current file and create a new file based on the configured rollover value.
rolloverSize *	Integer	-	Target file size, in KB, at which to trigger rollover. Applies to a Rollover write strategy. You can enter a value between 1 and 2147483647.
rolloverEvents *	Integer	-	Number of events or messages to accumulate before a rollover. Applies to a Rollover write strategy. You can enter a value between 1 and 2147483647.
rolloverTime *	Integer	-	Length of time, in milliseconds, after which to trigger a rollover. Applies to a Rollover write strategy. You can enter a value between 1 and 2147483647.
filesystemNameOverride	String	-	Overrides the default file system name provided in the connection. This file system name is used write to a file at run time. You can enter a string of up to 1,280 characters.
directoryOverride	String	-	Overrides the default directory path. The ADLS Gen2 directory path to write data to. If left blank, the default directory path is used. You can enter a string of up to 1,280 characters.
compressionFormat	String	-	Compression format to use before the streaming ingestion task writes data to the target file. Enter one of the following values: None GZIP BZIP2 DEFAULT1 Enter this value to use the Zlib format. DEFAULT2 Enter this value to use the Deflate format.
File Name/Expression	String	Yes	ADLS Gen2 file name or a regular expression. You can enter a string of up to 249 characters.
* Enter a value for at least one of the fields.

POST request example

To update a

streaming ingestion and replication

task with a flat file source and an ADLS Gen2 target, you might send a request similar to the following example:

{
  "name": "flatfile to adls",
  "description": "flatfile to adls",
  "runtimeId": "01000025000000000003",
  "locationId": "5sJ0JDyJyWLlrosS5qJjsQ",
  "currentVersion": "2",
  "messageFormat": "binary",
  "nodes": [
    {
      "name": "flatfile to adls_source",
      "type": "source",
      "connectionId": "012MGS0B00000000002N",
      "transformationType": "",
      "config": [
        {
          "key": "File",
          "value": "logfile"
        },
        {
          "key": "initialPosition",
          "value": "Current Time"
        },
        {
          "key": "rolloverPattern",
          "value": "test"
        },
        {
          "key": "tailingMode",
          "value": "Single file"
        }
      ]
    },
    {
      "name": "flatfile to adls_target",
      "type": "target",
      "connectionId": "012MGS0B00000000003D",
      "transformationType": "",
      "config": [
        {
          "key": "writeStrategy",
          "value": "Rollover"
        },
        {
          "key": "filesystemNameOverride",
          "value": "test"
        },
        {
          "key": "File Name/Expression",
          "value": "test"
        },
        {
          "key": "compressionFormat",
          "value": "NONE"
        },
        {
          "key": "directoryOverride",
          "value": "/test"
        },
        {
          "key": "interimDirectory",
          "value": "/home/agent/test"
        },
        {
          "key": "rolloverSize",
          "value": 1024
        },
        {
          "key": "rolloverEvents",
          "value": 100
        },
        {
          "key": "rolloverTime",
          "value": 300000
        }
      ]
    }
  ]
}

POST request

Download Guide

Watch

Comments

Cloud Data Integration Homepage