Amazon S3 V2 Connector

Back Next

Custom manifest file

You can read multiple files, which are of flat format type, from Amazon S3

and write data to a target

. To read multiple flat files, all files must be available in the same Amazon S3 bucket.

When you want to read from multiple sources in the Amazon S3 bucket, you must create a

.manifest

file that contains all the source files with the respective absolute path or directory path. You must specify the

.manifest

file name in the following format:

<file_name>.manifest

For example, the

.manifest

file contains source files in the following format:


{
    "fileLocations": [
        {
            "URIs": [
                "dir1/dir2/dir3/file_1.csv",
                "dir1/dir2/dir3/file_2.csv",
                "dir1/file_3.csv"
            ]
        },
        {
            "URIPrefixes": [
                "dir1/dir2/dir3/",
                "dir1/dir2/dir4/"
            ]
        },
        {
            "WildcardURIs": [
                "dir1/dir2/dir3/*.csv"
            ]
        }
    ]
}

The custom manifest file contains the following tags:

URIs. Specify the path for the files relative to the bucket name.

URIPrefixes. Specify the path for the directory relative to the bucket name.

WildcardURIs. Specify an asterisk (*) wildcard in the file name, which are of flat format type, to fetch files from the Amazon S3 bucket. Specify the asterisk (*) wildcard to fetch all the files or only the files that match the name pattern.

You can specify URIs, URIPrefixes, WildcardURIs, or all sections within fileLocations in the

.manifest

file.

You cannot use the wildcard characters to specify folder names. For example, { "WildcardURIs": [ "multiread_wildcard/dir1*/", "multiread_wildcard/*/" ] }.

The

Data Preview

tab displays the data of the first file available in the URI specified in the

.manifest

file. If the URI section is empty, the first file in the folder specified in URIPrefixes is displayed.

Rename Saved Search

Table of Contents

Amazon S3 V2 Connector

Amazon S3 V2 Connector

Custom manifest file

Custom manifest file