Table of Contents

Search

  1. Preface
  2. Introduction to Data Engineering Streaming
  3. Data Engineering Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Transformation in Streaming Mappings
  8. Window Transformation
  9. Appendix A: Connections
  10. Appendix B: Monitoring REST API Reference
  11. Appendix C: Sample Files

Advanced Properties

Advanced Properties

The Developer tool displays the advanced properties for Amazon S3 targets in the Input transformation in the
Write
view.
You can configure the following advanced properties for Amazon S3 targets:
Property
Description
Operation Type
Specifies the type of data object operation.
This is a read-only property.
Overwrite File(s) If Exists
Not applicable for streaming mappings.
Folder Path
Bucket name or folder path of the Amazon S3 source file that you want to write to.
If applicable, include the folder name that contains the target file in the
<bucket_name>/<folder_name>
format.
If you do not provide the bucket name and specify the folder path starting with a slash (/) in the
/<folder_name>
format, the folder path appends with the folder path that you specified in the connection properties.
For example, if you specify the
<my_bucket1>/<dir1>
folder path in the connection property and
/<dir2>
folder path in this property, the folder path appends with the folder path that you specified in the connection properties in
<my_bucket1>/<dir1>/<dir2>
format.
If you specify the
<my_bucket1>/<dir1>
folder path in the connection property and
<my_bucket2>/<dir2>
folder path in this property, the Data Integration Service writes the file in the
<my_bucket2>/<dir2>
folder path that you specify in this property.
File Name
Name of the Amazon S3 target file that you want to write to. The file specified at run time overrides the file specified while creating a connection.
Encryption Type
Method you want to use to encrypt data.
Select one of the following values:
  • None
  • Server Side Encryption
Client side encryption and server side encryption with KMS are not applicable for streaming mappings.
Staging Directory
Not applicable for streaming mappings.
File Merge
Merges the target files into a single file.
Not applicable for streaming mappings.
Hadoop Performance Tuning Options
Not applicable for streaming mappings.
Compression Format
Compresses data when you write data to Amazon S3.
You can compress the data in the following formats:
  • None
  • Bzip2
  • Deflate
  • Gzip
  • Lzo
  • Snappy
  • Zlib
Default is
None
.
Object Tags
Not applicable for streaming mappings.
TransferManager Thread Pool Size
The number of threads to write data in parallel. Default is 10.
When the file size is more than 5 MB, you can configure multipart upload to upload object in multiple parts in parallel. If you set the value of the
TransferManager Thread Pool Size
to greater than 50, the value reverts to 50.
Not applicable for streaming mappings.
Part Size
The part size in bytes to upload an Amazon S3 object. Default is 5 MB.
Not applicable for streaming mappings.
Temporary Credential Duration
The time duration during which an IAM user can use the dynamically generated temporarily credentials to access the AWS resource. Enter the time duration in seconds.
Default is 900 seconds.
If you require more than 900 seconds, you can set the time duration maximum up to 12 hours in the AWS console and then enter the same time duration in this property.
Stream Rollover Size in GB
Optional. Target file size, in gigabytes (GB), at which to trigger rollover. A value of zero (0) means that the target file does not roll over based on size. Default is 1 GB.
Stream Rollover Time in Hours
Optional. Length of time, in hours, for a target file to roll over. After the time period has elapsed, the target file rolls over. A value of zero (0) means that the target file does not roll over based on time. Default is 1 hour.
Interim Directory
The active directory location of the complex file target. This directory stages all the files currently in open state. When the stream rollover condition is met, the files are moved from the interim directory to the target directory.

0 COMMENTS

We’d like to hear from you!