A Streamer splits a large source document into smaller portions that a transformation can process separately. Streamers are useful in transformations that process very large inputs, such as multi-gigabyte data streams. A Streamer can have a buffer input or file input.
A Streamer offers the following advantages:
The transformation parses each source segment when it is available, instead of waiting until it receives entire source.
The transformation has reduced memory requirements.
For example, a Streamer input stream might contain stock market transaction data. The stream transmits to a server continually over the course of the entire trading day. A Script with a Streamer processes each transaction when it arrives, instead of waiting until the end of the day.
In another example, you receive a large source file over an FTP connection. By using a Streamer, the Script can process the file before it is completely received.
The Data Processor transformation provides the following kinds of Streamers:
Streamer. Processes large text inputs.
XmlStreamer. Process large XML inputs.
Streamers are runnable components. Define the
component at the global level of the Script, and set it as the startup component of the transformation. The Streamer functions by splitting the input into segments and passing them to other runnable components, which can be Parsers, Mappers, or Serializers.