Table of Contents

Search

  1. Preface
  2. Introduction to Data Engineering Streaming
  3. Data Engineering Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Appendix A: Connections
  9. Appendix B: Monitoring REST API Reference
  10. Appendix C: Sample Files

Temporary Directory for the Complex File Target

Temporary Directory for the Complex File Target

At run time, the Data Integration Service creates a temporary directory in the location that you set for the
File Directory
property that you configure for the complex file targets in a streaming mapping. A temporary directory is created for each mapping. A temporary directory contains the target files to which the data is currently written.
Based on the rollover limit, data is written to the target file in the temporary directory. After a target file reaches the specified rollover limit, the target file is closed and moved to the specified target file directory from the temporary directory.
For example, if you set the
File Directory
property to
hdfs://demo.domain.com:8020/tmp/batch/
, you can find the target file to which data is currently written in the following directory:
hdfs://demo.domain.com:8020/tmp/batch/active/586bdfd67170008d125f52f68c1f02/
. The
hdfs://demo.domain.com:8020/tmp/batch/
directory contains the target files that are moved after the rollover limit is reached.
If a mapping fails, all the target files are closed and moved to the specified target file directory from the temporary directory, and the temporary directory will be deleted. In the preceding example, the temporary directory
586bdfd67170008d125f52f68c1f02
will be deleted.
If you use Data Engineering Integration to process batch data, you can use the batch complex files that are created in the file directory as the source objects in Data Engineering Integration.

0 COMMENTS

We’d like to hear from you!