REST API Reference

Back Next

Kafka sources and targets

When the source and target connection of the task is Kafka, you can include the following fields and key-value pairs in the config array of the source and target connection:

Key	Type	Required	Description
Topic	String	Yes	Kafka source topic name or a Java supported regular expression for the Kafka source topic name pattern to read the events from. Enter a string of up to 249 characters.
consumerProperties	String	-	Provide a comma-separated list of optional consumer configuration properties. Specify the values as key-value pairs. For example, key1=value1, key2=value2 . You can enter a string of up to 4000 characters.
producerProperties	String	-	The configuration properties for the producer. Provide a comma-separated list and specify the values as key-value pairs. You can enter a string of up to 4000 characters.
mdFetchTimeout	Integer	-	The time after which the metadata is not fetched. Enter a value between 1 and 2147483647.
batchSize	Integer	-	The batch size of the events after which a streaming ingestion task writes data to the target. Enter a value between 1 and 2147483647.
Topic Name/Expression	String	Yes	Kafka topic name or a Java supported regular expression for the Kafka topic name pattern. You can enter a string of up to 249 characters.

POST request example

To update a

streaming ingestion and replication

task with a Kafka source and target, you might send a request similar to the following example:

POST <serverUrl>/sisvc/restapi/v1/UpdateEntity/Documents('<document ID>')
Content-Type: application/json
Accept: application/json
IDS-SESSION-ID: <SessionId>

{
  "name": "kafka to kafka",
  "description": "kafka to kafka",
  "runtimeId": "01000025000000000003",
  "locationId": "5sJ0JDyJyWLlrosS5qJjsQ",
  "currentVersion": "2",
  "messageFormat": "binary",
  "nodes": [
    {
      "name": "kafka to kafka_source",
      "type": "source",
      "connectionId": "012MGS0B000000000002",
      "transformationType": "",
      "config": [
        {
          "key": "consumerProperties",
          "value": "key=value"
        },
        {
          "key": "Topic",
          "value": "test"
        }
      ]
    },
    {
      "name": "kafka to kafka_target",
      "type": "target",
      "connectionId": "012MGS0B000000000002",
      "transformationType": "",
      "config": [
        {
          "key": "producerProperties",
          "value": "key=value"
        },
        {
          "key": "mdFetchTimeout",
          "value": 5000
        },
        {
          "key": "batchSize",
          "value": 1048576
        },
        {
          "key": "Topic Name/Expression",
          "value": "test"
        }
      ]
    }
  ],
  "edges": [
    {
      "from": "kafka to kafka_source",
      "to": "kafka to kafka_target"
    }
  ]
}

The response returns only the fields that you entered in the request. If the request is unsuccessful, the response includes a reason for the failure.

POST response example

If the request is successful, you might receive a response similar to the following example in a

Success

node:

{
  "Success": {
    "name": "kafka to kafka",
    "description": "kafka to kafka",
    "runtimeId": "01000025000000000003",
    "locationId": "5sJ0JDyJyWLlrosS5qJjsQ",
    "currentVersion": "2",
    "messageFormat": "binary",
    "nodes": [
      {
        "name": "kafka to kafka_source",
        "type": "source",
        "connectionId": "012MGS0B000000000002",
        "transformationType": "",
        "config": [
          {
            "key": "consumerProperties",
            "value": "key=value"
          },
          {
            "key": "Topic",
            "value": "test"
          }
        ]
      },
      {
        "name": "kafka to kafka_target",
        "type": "target",
        "connectionId": "012MGS0B000000000002",
        "transformationType": "",
        "config": [
          {
            "key": "producerProperties",
            "value": "key=value"
          },
          {
            "key": "mdFetchTimeout",
            "value": 5000
          },
          {
            "key": "batchSize",
            "value": 1048576
          },
          {
            "key": "Topic Name/Expression",
            "value": "test"
          }
        ]
      }
    ],
    "edges": [
      {
        "from": "kafka to kafka_source",
        "to": "kafka to kafka_target"
      }
    ]
  }
}

Rename Saved Search

Table of Contents

REST API Reference

REST API Reference

Kafka sources and targets

Kafka sources and targets

POST request example

POST response example