Table of Contents

Search

  1. Preface
  2. Introduction to Amazon S3 V2 Connector
  3. Connections for Amazon S3 V2
  4. Amazon S3 V2 sources and targets
  5. Mappings and mapping tasks with Amazon S3 V2
  6. Migrating a mapping
  7. Upgrading to Amazon S3 V2 Connector
  8. Data type reference
  9. Troubleshooting

Amazon S3 V2 Connector

Amazon S3 V2 Connector

Amazon S3 V2 targets in mappings

Amazon S3 V2 targets in mappings

In a mapping, you can configure a Target transformation to represent an Amazon S3 V2 object as the target to write data to Amazon S3.
Specify the name and description of the Amazon S3 V2 target. Configure the Amazon S3 V2 target and advanced properties for the target object.
The following table describes the Amazon S3 V2 target properties that you can configure in a Target transformation:
Property
Description
Connection
Name of the Amazon S3 V2 target connection. Select a target connection or click
New Parameter
to define a new parameter for the target connection.
If you want to overwrite the parameter at runtime, select the
Allow parameter to be overridden at run time
option when you create a parameter. When the task runs, the agent uses the parameters from the file that you specify in the task advanced session properties.
Target Type
Target type. Select one of the following types:
  • Single Object
  • Parameter: Select
    Parameter
    to define the target type when you configure the mapping task.
Object
Name of the target object.
You can select an existing object or create an object at runtime. When you create an object at runtime, enter a name and the path for the target object.
Parameter
Select an existing parameter for the source object or click
New Parameter
to define a new parameter for the target object. The Parameter property appears only if you select Parameter as the target type.
If you want to overwrite the parameter at runtime, select the
Allow parameter to be overridden at run time
option when you create a parameter. When the task runs, the agent uses the parameters from the file that you specify in the task advanced session properties.
Create Target
Creates a target.
Enter a name and path for the target object. You can use parameters defined in a parameter file in the target name.
For more information, see Rules and guidelines for creating a target.
Format
Specifies the file format that the Amazon S3 V2 Connector uses to write data Amazon S3.
You can select the following file format types:
  • None
    1
  • Flat
  • Avro
  • ORC
  • Parquet
  • JSON
    2
  • Delta
    1
Default is
None
. If you select
None
is as the format type, the Secure Agent writes data to Amazon S3 files in binary format.
Open the
Formatting Options
dialog box to define the format of the file.
For more information about format options, see File formatting options.
Operation
Type of the target operation.
You can perform only insert operation on an Amazon S3 V2 target.
1
Doesn't apply to mappings in advanced mode.
2
Applies only to mappings in advanced mode.
The following table describes the Amazon S3 V2 advanced target properties that you can configure in a Target transformation:
Property
Description
Overwrite File(s) If Exists
Overwrites an existing target file.
Default is true. For more information about overwriting the existing files, see Overwriting existing files.
Folder Path
Bucket name or folder path where you want to write the Amazon S3 target file. The path that you enter here overrides the path specified for the target configured to create at runtime.
If applicable, include the folder name that contains the target file in the
<bucket_name>/<folder_name>
format.
If you do not provide the bucket name and specify the folder path starting with a slash (/) in the
/<folder_name>
format, the folder path appends with the folder path that you specified in the connection properties.
For example, if you specify the
/<dir2>
folder path in this property and
<my_bucket1>/<dir1>
folder path in the connection property, the folder path appends with the folder path that you specified in the connection properties in
<my_bucket1>/<dir1>/<dir2>
format.
If you specify the
<my_bucket1>/<dir1>
folder path in the connection property and
<my_bucket2>/<dir2>
folder path in this property, the Secure Agent writes the file in the
<my_bucket2>/<dir2>
folder path that you specify in this property.
File Name
Creates a new file name or overwrites an existing target file name.
Encryption Type
Method you want to use to encrypt data.
Select one of the following encryption types:
  • None
  • Client Side Encryption
    1
  • Server Side Encryption
  • Server Side Encryption with KMS
  • Informatica Encryption
Default is
None
.
For more information about the encryption type, see Data encryption in Amazon S3 V2 targets.
Staging Directory
1
Enter the path of the local staging directory.
Ensure that the user has write permissions on the directory. In addition, ensure that there is sufficient space to enable staging of the entire file. Default staging directory is the
/temp
directory on the machine that hosts the Secure Agent.
When you specify the directory path, the Secure Agent create folders depending on the number of partitions that you specify in the following format:
InfaS3Staging<00/11><timestamp>_<partition number>
where, 00 represents read operation and 11 represents write operation.
For example,
InfaS3Staging000703115851268912800_0
The temporary files are created within the new directory.
The staging directory target property does not apply to Avro, ORC, Parquet, and Delta files.
File Merge
This property is not applicable for Amazon S3 V2 Connector.
Hadoop Performance Tuning Options
This property is not applicable for Amazon S3 V2 Connector.
Compression Format
Compresses data when you write data to Amazon S3.
You can compress the data in the following formats:
  • None
  • Bzip2
    2
  • Deflate
  • Gzip
  • Lzo
  • Snappy
  • Zlib
Default is None.
Amazon S3 V2 Connector does not support the Lzo compression format even though the option appears in this property.
For more information about the compression format, see Data compression in Amazon S3 V2 sources and targets.
Object Tags
The key value pairs to add single or multiple tags to the objects stored on the Amazon S3 bucket.
You can either enter the key value pairs or specify the file path that contains the key value pairs.
Use this property when you run a mapping to write a file of flat format type. For more information about the object tags, see Object tag.
This property applies only to flat files.
TransferManager Thread Pool Size
1
The number of threads to write data in parallel.
Default is 10. Use this property when you run a mapping to write a file of flat format type.
Amazon S3 V2 Connector uses the
AWS TransferManager API
to upload a large object in multiple parts to Amazon S3.
When the file size is more than 5 MB, you can configure multipart upload to upload object in multiple parts in parallel. If you set the value of
TransferManager Thread Pool Size
to greater than 50, the value reverts to 50.
This property applies only to flat files.
Merge Partition Files
1
Determines whether the Secure Agent must merge the number of partition files as a single file or maintain separate files based on the number of partitions specified to write data to the Amazon S3 V2 targets.
This property applies only to flat files.
Temporary Credential Duration
The time duration during which an IAM user can use the dynamically generated temporarily credentials to access the AWS resource. Enter the time duration in seconds.
Default is 900 seconds.
If you require more than 900 seconds, you can set the time duration maximum up to 12 hours in the AWS console and then enter the same time duration in this property.
Part Size
1
Uploads the part size of an Amazon S3 object in bytes.
Default is 5 MB. Use this property when you run a mapping to write a file of flat format type.
This property applies only to flat files.
Forward Rejected Rows
This property is not applicable for Amazon S3 V2 Connector.
1
Doesn't apply to mappings in advanced mode.
2
Applies only to mappings in advanced mode.
When you create a mapping and the column name in the Amazon S3 source or target object contains special characters, the Secure Agent replaces the special characters with an underscore
(_)
and the mapping fails.

0 COMMENTS

We’d like to hear from you!