Amazon S3 Connector Guide

Amazon S3 Connector Guide

Writing Content with Event Targets

Writing Content with Event Targets

When you publish a connection that includes an event target definition for S3, you create a service that you can call from a process to generate S3 objects.

Writer Event Targets

StoreS3ObjectRequest
is the process object that contains the target object parameters and contents, as shown here. This is the input to the event target:
<StoreS3ObjectRequest> <!-- create S3 object metadata --> <s3ObjectParameters> <!-- object key, is required --> <awsS3Key>test.txt</awsS3Key> <!-- content type, by default text/plain type is used --> <contentType>text/plain</contentType> <!-- content control --> <contentControl>…</contentControl> <!-- content disposition which is a default file name (similar to Content-Disposition HTTP header) --> <contentDisposition>…</contentDisposition> <!-- content encoding, by default this property is empty --> <contentEncoding>…</contentEncoding> <!-- last modified timestamp in milliseconds --> <lastModified>…</lastModified> <!-- storage class (STANDARD or REDUCED_REDUNDANCY), can be used to overwrite the same attribute of the event target --> <storageClass>…</storageClass> <!-- canned acl (Private, PublicRead, PublicReadWrite, AuthenticatedRead, LogDeliveryWrite, BucketOwnerRead, BucketOwnerFullControl). --> <cannedAcl>…</cannedAcl> <!-- a well-constructed Amazon S3 Access Control List xml string. --> <s3Acl>…</s3Acl> </s3ObjectParameters> <!-- content format (PlainText|Binary|Attachment|XML|JSON) depending on what format you use the content should be provided differently --> <format>PlainText|Binary|Attachment|XML|JSON</format> <!-- this field can be used only if you set format to PlainText or Binary. if you use PlainText format this field should contain some textual information that will be written as is to the bucket; if you use Binary format here you should provide a Base64 encoded string that will be decoded and written to the target bucket --> <content>Hello World</content> <!-- if you chose the XML or JSON format you should use either the object or objects field to provide one or several process objects that should be converted to XML/JSON content --> <object>some process object</object> <objects>a list of process objects</objects> <!-- optionally you can provide objectName and listName attribute values, details see below this example --> <objectName>user</objectName> <listName>users</listName> </StoreS3ObjectRequest >
For details on the Access Control List (ACL) parameters, see the Amazon documentation on Access Control List Overview and CannedAccessControlList.
The only required parameters are:
  • AwsS3Key
  • format
You can omit all other parameters. Depending on the format you select, the parameters vary.
StoreS3ObjectResponse
is the process object returned by the Writer event target similar to the following:
<StoreS3ObjectResponse> <!-- S3 object version (if available) --> <awsS3VersionId/> <!-- S3 object ETag --> <awsS3ETag>b10a8db164e0754105b7a99be72e3fe5</awsS3ETag> </StoreS3ObjectResponse>

XML/JSON Format

When you use XML/JSON format, the S3 connection expects an
object
(a single process object) or
objects
(a list of process objects). If you pass in a single process object using the objects field, the result is a list with a single item. If you pass in the same object using object fields, the result is a single XML/JSON object (not a list).
You can use the optional parameters,
objectName
and
listName
, to change the generated XML and JSON content as shown below.
XML
: When you serialize a list of
user
process objects to XML format, the output is similar to:
<objects> <object> <firstName>Bob</firstName> <lastName>Smith</lastName> <phone>111122-222</phone> </object> <object> <firstName>Bob2</firstName> <lastName>Smith2</lastName> <phone>111122-222</phone> </object> </objects>
The root elements have generic names (object and objects) by default because Process Designer does not have information about any object list or individual object names.
If you need to change the root element names, you can use the
objectName
and
listName
parameters to specify element names.
JSON:
To serialize the same objects list to JSON format, you can use the
objectName
and
listName
parameters to add a wrapper JSON object to the generated result. Without these parameters, the output is a valid JSON array:
[{ "firstName": "Bob", "lastName": "Smith", "phone": "1111-222-333" }, { "firstName": "Bob2", "lastName": "Smith", "phone": "1111-222-333" }]
You can also serialize the same process objects list as a JSON object as a single array by adding the "users" wrapper in the
listName
parameter:
{ “users”: [{ "firstName": "Bob", "lastName": "Smith", "phone": "1111-222-333" }, { "firstName": "Bob2", "lastName": "Smith", "phone": "1111-222-333" }] }
As shown above, the JSON object now contains a single JSON object wrapper with a single field that contains the list
user
objects.
To wrap a single object in the same way, use the
objectName
parameter instead.

Attachment Format

If you use the Attachment format, the approach is different. Instead of providing content as part of the
StoreS3ObjectRequest
object,, you can use a separate
S3Writer
service call parameter,
attachment
.

Delimited Content Writer Event Targets

You can serialize both built-in delimited content process objects and a list of custom process objects.
When you publish a connection that contains a delimited content event target, Process Designer creates the following process objects:
  • StoreS3DelimitedContentRequest
  • StoreS3DelimitedContentResponse
Whether you serialize the
S3DelimitedContent
process object or a list of custom process objects, the request and response are nearly identical.
For example:
You can serialize both built-in delimited content process objects and a list of custom process objects.
<StoreS3DelimitedContentRequest> <!-- create S3 object metadata --> <s3ObjectParameters> <!-- object key, is required --> <awsS3Key>test.csv</awsS3Key> <!-- other optional parameters (the same as S3 Writer request parameters) --> </s3ObjectParameters> <!—delimiter char; overwrites the same setting of the event target --> <delimiter>;</delimiter> <!-- text qualifier char; overwrites the same setting of the event target --> <textQualifier>”</textQualifier> <!-- whether to skip headers; overwrites the same setting of the event target --> <skipHeaders>false</skipHeaders> <!-- built-in S3DelimitedContent process object model, in this case not used --> <delimitedContentObject/> <!-- list of custom headers --> <header> <!-- header name (custom object field name) --> <name>name</name> <!-- header index (column position index) --> <fieldIndex>1</fieldIndex> </header> <header> <name>address</name> <fieldIndex>2</fieldIndex> </header> <header> <name>phone</name> <fieldIndex>3</fieldIndex> </header> <!-- list of custom process objects --> <customObjects> <id>1</id> <name>Bill</name> <phone>123-11-22</phone> <address>Lviv</address> </customObjects> <customObjects> <id>2</id> <name>Bob</name> <phone>222-333-222</phone> <address>London</address> </customObjects> </StoreS3DelimitedContentRequest>
Note
: The object metadata parameters are the same as those used in the S3 Writer. If needed, you can overwrite some of the event target attributes (delimiter, text qualifier, and skip headers).

Handling Headers with Custom Process Objects

Process Designer can automatically serialize simple objects but skips complex fields (such as references and object lists) when using the custom process objects. The generated file includes a list of headers using the first process object's simple field names. In this case, you should leave the
delimitedContentObject
field (see above) empty. During serialization, Process Designer generates a list of headers using the first process object's simple field names. Sometimes this method is useful but it has several disadvantages:
  • If the first object does not contain optional fields, these fields are ignored even if they are provided in all other objects.
  • The order of fields in the resulting object is not defined because the process object's fields do not rely on any specific order.
  • You cannot skip unnecessary fields.
You can eliminate these disadvantages if you provide a set of custom headers in a request object. Process Designer then uses the headers to generate delimited records with only the specified fields in the specified order.
If you work with the built-in process objects, when you define the request object, you can include the
S3DelimitedContent
process object in the request. The generated delimited content then matches the headers and records described by this model object.
The
s3ObjectInfo
and
localFileInfo
fields in the
S3DelimitedContent
object are ignored during serialization so you can leave them empty if you create a new object yourself or retain them if you received the delimited content process object from somewhere and these fields are set

Serialization Results

Whatever approach you take, after each file is processed, the results display:
  • Status of the operation.
  • Optional message string.
  • Two counters with the number of processed records and number of records successfully written to S3 object content.
  • Object metadata for the new S3 objects.

StoreS3DelimitedContentResponse

The response process object contains:
<StoreS3DelimitedContentResponse> <!-- true if success, false in case of error --> <success>true</success> <!-- textual message that may contain some warnings or errors --> <message>…</message> <!-- number of processed records (number of records that are provided in request) --> <processedRecordsCount>3</processedRecordsCount> <!-- number of successfully serialized and written to S3 object content records --> <writtenRecordsCount>3</writtenRecordsCount> <!-- created S3 object information --> <s3ObjectInfo> <!-- S3 object version (if available) --> <awsS3VersionId/> <!-- S3 object ETag --> <awsS3ETag>ae784bc7446c154e2f802c8440a530a0</awsS3ETag> </s3ObjectInfo> </StoreS3DelimitedContentResponse>