REST API Reference

Back Next

Connection configuration for tasks with Amazon S3 as a target

When the target connection of the task target is Amazon S3, you can include the following fields and key-value pairs in the config array of the target connection:

Key	Type	Required	Description
partitionTime	String	-	The time interval according to which the streaming ingestion task creates partitions in the Amazon S3 bucket. Enter one of the following values: None 5min 10min 15min 20min 30min 1hr 1day
minUploadPartSize	Integer	-	Minimum part size when uploading a large file as a set of multiple independent parts, in megabytes. Use this property to tune the file load to Amazon S3. You can enter a value between 50 and 5120.
multipartUploadThreshold	Integer	-	Multipart threshold when uploading objects in multiple parts in parallel. You can enter a value between 50 and 5120.
Object Name/Expression	String	Yes	Amazon S3 target file name or a regular expression for the Amazon S3 file name pattern.

POST request example

To update a

streaming ingestion

task with a flat file source and an Amazon S3 as target, you might send a request similar to the following example:

{
  "name": "flatfile to amazon S3",
  "description": "flatfile to amazon S3",
  "runtimeId": "01000025000000000003",
  "locationId": "5sJ0JDyJyWLlrosS5qJjsQ",
  "currentVersion": "2",
  "messageFormat": "binary",
  "nodes": [
    {
      "name": "flatfile to amazon S3_source",
      "type": "source",
      "connectionId": "012MGS0B00000000002N",
      "transformationType": "",
      "config": [
        {
          "key": "File",
          "value": "logfile"
        },
        {
          "key": "initialPosition",
          "value": "Current Time"
        },
        {
          "key": "rolloverPattern",
          "value": "test"
        },
        {
          "key": "tailingMode",
          "value": "Single file"
        }
      ]
    },
    {
      "name": "flatfile to amazon S3_target",
      "type": "target",
      "connectionId": "012MGS0B0000000000I7",
      "transformationType": "",
      "config": [
        {
          "key": "partitionTime",
          "value": "None"
        },
        {
          "key": "minUploadPartSize",
          "value": 5120
        },
        {
          "key": "multipartUploadThreshold",
          "value": 5120
        },
        {
          "key": "Object Name/Expression",
          "value": "test"
        }
      ]
    }
  ],
  "edges": [
    {
      "from": "flatfile to amazon S3_source",
      "to": "flatfile to amazon S3_target"
    }
  ]
}

Rename Saved Search

Table of Contents

REST API Reference

REST API Reference

Connection configuration for tasks with Amazon S3 as a target

Connection configuration for tasks with Amazon S3 as a target

POST request example