Using PowerCenter to Process Flat Files in Real Time

Back Next

Overview

By default, a PowerCenter session reads and writes bulk data at scheduled intervals. If you process flat file data based on a time schedule, use sessions that process multiple flat files in bulk. When you configure a PowerCenter session for real-time processing, the session reads, processes, and writes data to targets continuously. If you process flat file data based on data arrival, use real-time sessions.

You can use a session that is not configured for real-time processing to read a single flat file when it arrives. However, session processing based on flat file arrival can run into the following scalability issues:

If a workflow is trigged with each arrival of a flat file and hundreds of files arrive every minute, you might encounter a high number of concurrent workflows that can cause performance issues.

If a single session processes one file at a time, and you need to process thousands of flat files daily, the time that it takes to reestablish the connection for each session might cause performance issues.

To solve the scalability issues, consider the following solutions to process flat files in real time or near real time:

Run sessions that process multiple files at regular intervals.

Use a PowerCenter file list or use B2B Data Exchange with delayed event processing.

Run a single real-time session that reads, processes, and writes flat file data to targets continuously. Real-time sessions require messages or message queues as the real-time source. Real-time sessions must read flat file sources midstream in the pipeline.

Use real-time processing or use B2B Data Exchange with real-time processing.

Using PowerCenter to Process Flat Files in Real Time

Benefits and Limitations of Flat File Processing Solutions

Download Guide

Watch

Comments

Communities

Knowledge Base

Success Portal