When a mapping that is enabled for partitioning reads from a flat file source, the Data Integration Service can use multiple threads to read the file source.
The Data Integration Service can create partitions for the following flat file source types:
Direct file
Indirect file
Directory of files
Command
File or directory of files in Hadoop Distributed File System (HDFS)
When the Data Integration Service uses multiple threads to read a file source, it creates multiple concurrent connections to the source. By default, the Data Integration Service does not preserve row order because it does not read the rows in the file or file list sequentially. To preserve row order when multiple threads read from a single file source, configure concurrent read partitioning.
When the Data Integration Service uses multiple threads to read a direct file, it creates multiple reader threads to read the file concurrently.
When the Data Integration Service uses multiple threads to read an indirect file or a directory of files, it creates multiple reader threads to read the files in the list or directory concurrently. The Data Integration Service might use multiple threads to read a single file. Or, the Data Integration Service might use a single thread to read multiple files in the list or directory.