REST API Reference

Back Next

Amazon S3 targets

When the target connection of the task target is Amazon S3, you can include the following fields and key-value pairs in the config array of the target connection:

Key	Type	Required	Description
partitionTime	String	-	The time interval according to which the streaming ingestion task creates partitions in the Amazon S3 bucket. Enter one of the following values: None 5min 10min 15min 20min 30min 1hr 1day
minUploadPartSize	Integer	-	Minimum part size when uploading a large file as a set of multiple independent parts, in megabytes. Use this property to tune the file load to Amazon S3. You can enter a value between 50 and 5120.
multipartUploadThreshold	Integer	-	Multipart threshold when uploading objects in multiple parts in parallel. You can enter a value between 50 and 5120.
Object Name/Expression	String	Yes	Amazon S3 target file name or a regular expression for the Amazon S3 file name pattern.

POST request example

To update a

streaming ingestion and replication

task with a flat file source and an Amazon S3 as target, you might send a request similar to the following example:

POST <serverUrl>/sisvc/restapi/v1/UpdateEntity/Documents('<document ID>')
Content-Type: application/json
Accept: application/json
IDS-SESSION-ID: <SessionId>

{
  "name": "flatfile to amazon S3",
  "description": "flatfile to amazon S3",
  "runtimeId": "01000025000000000003",
  "locationId": "5sJ0JDyJyWLlrosS5qJjsQ",
  "currentVersion": "2",
  "messageFormat": "binary",
  "nodes": [
    {
      "name": "flatfile to amazon S3_source",
      "type": "source",
      "connectionId": "012MGS0B00000000002N",
      "transformationType": "",
      "config": [
        {
          "key": "File",
          "value": "logfile"
        },
        {
          "key": "initialPosition",
          "value": "Current Time"
        },
        {
          "key": "rolloverPattern",
          "value": "test"
        },
        {
          "key": "tailingMode",
          "value": "Single file"
        }
      ]
    },
    {
      "name": "flatfile to amazon S3_target",
      "type": "target",
      "connectionId": "012MGS0B0000000000I7",
      "transformationType": "",
      "config": [
        {
          "key": "partitionTime",
          "value": "None"
        },
        {
          "key": "minUploadPartSize",
          "value": 5120
        },
        {
          "key": "multipartUploadThreshold",
          "value": 5120
        },
        {
          "key": "Object Name/Expression",
          "value": "test"
        }
      ]
    }
  ],
  "edges": [
    {
      "from": "flatfile to amazon S3_source",
      "to": "flatfile to amazon S3_target"
    }
  ]
}

The response returns only the fields that you entered in the request. If the request is unsuccessful, the response includes a reason for the failure.

POST response example

If the request is successful, you might receive a response similar to the following example in the

Success

node:

{
    "Success": {
    "name": "flatfile to amazon S3",
    "description": "flatfile to amazon S3",
    "runtimeId": "01000025000000000003",
    "locationId": "5sJ0JDyJyWLlrosS5qJjsQ",
    "currentVersion": "2",
    "messageFormat": "binary",
    "nodes": [
      {
        "name": "flatfile to amazon S3_source",
        "type": "source",
        "connectionId": "012MGS0B00000000002N",
        "transformationType": "",
        "config": [
          {
            "key": "File",
            "value": "logfile"
          },
          {
            "key": "initialPosition",
            "value": "Current Time"
          },
          {
            "key": "rolloverPattern",
            "value": "test"
          },
          {
            "key": "tailingMode",
            "value": "Single file"
          }
        ]
      },
      {
        "name": "flatfile to amazon S3_target",
        "type": "target",
        "connectionId": "012MGS0B0000000000I7",
        "transformationType": "",
        "config": [
          {
            "key": "partitionTime",
            "value": "None"
          },
          {
            "key": "minUploadPartSize",
            "value": 5120
          },
          {
            "key": "multipartUploadThreshold",
            "value": 5120
          },
          {
            "key": "Object Name/Expression",
            "value": "test"
          }
        ]
      }
    ],
    "edges": [
      {
        "from": "flatfile to amazon S3_source",
        "to": "flatfile to amazon S3_target"
      }
    ]
  }
}

Rename Saved Search

Table of Contents

REST API Reference

REST API Reference

Amazon S3 targets

Amazon S3 targets

POST request example

POST response example