Table of Contents

Search

Overwriting Existing Files

Overwriting Existing Files

You can choose to overwrite the existing files.
Select the
Overwrite File(s) If Exists
option in the Amazon S3 data object write operation properties to overwrite the existing files. By default, the value of the
Overwrite File(s) If Exists
check box is true.
If you select the
Overwrite File(s) If Exists
option, the Data Integration Service deletes the existing files with same file name and creates a new files with the same file name in the target directory.
If you do not select the
Overwrite File(s) If Exists
option, the Data Integration Service does not delete the existing files in the target directory. The Data Integration Service adds time stamp at the end of each target file name in the following format:
YYYYMMDD_HHMMSS_millisecond
. For example, the Data Integration Service renames the target file name in the following format:
output.txt-20170413_220348_164-m-00000
.
If you select the
Overwrite File(s) If Exists
option on the Spark engine, the Data Integration Service splits the existing files into multiple files with same file name. Then the Data Integration Service deletes the split files and creates new files in the target directory.
When you select the
Overwrite File(s) If Exists
option to overwrite an Avro file on the Spark engine, the Data Integration Service overwrites the existing file and appends
_avro
to the folder name. For example,
targetfile_avro


Updated July 30, 2020