Table of Contents

Search

  1. Preface
  2. Introduction to Data Transformation
  3. Data Processor Transformation
  4. Wizard Input and Output Formats
  5. Relational Input and Output
  6. Using the IntelliScript Editor
  7. XMap
  8. Libraries
  9. Schema Object
  10. Command Line Interface
  11. Scripts
  12. Parsers
  13. Script Ports
  14. Document Processors
  15. Formats
  16. Data Holders
  17. Anchors
  18. Transformers
  19. Actions
  20. Serializers
  21. Mappers
  22. Locators, Keys, and Indexing
  23. Streamers
  24. Validators, Notifications, and Failure Handling
  25. Validation Rules
  26. Custom Script Components

Data Transformation User Guide

Data Transformation User Guide

Streamers Overview

Streamers Overview

A Streamer splits a large source document into smaller portions that a transformation can process separately. Streamers are useful in transformations that process very large inputs, such as multi-gigabyte data streams. A Streamer can have a buffer input or file input.
A Streamer offers the following advantages:
  • The transformation parses each source segment when it is available, instead of waiting until it receives entire source.
  • The transformation has reduced memory requirements.
For example, a Streamer input stream might contain stock market transaction data. The stream transmits to a server continually over the course of the entire trading day. A Script with a Streamer processes each transaction when it arrives, instead of waiting until the end of the day.
In another example, you receive a large source file over an FTP connection. By using a Streamer, the Script can process the file before it is completely received.
The Data Processor transformation provides the following kinds of Streamers:
  • Streamer. Processes large text inputs.
  • XmlStreamer. Process large XML inputs.
Streamers are runnable components. Define the
Streamer
or
XmlStreamer
component at the global level of the Script, and set it as the startup component of the transformation. The Streamer functions by splitting the input into segments and passing them to other runnable components, which can be Parsers, Mappers, or Serializers.


Updated September 26, 2018