Table of Contents

Search

  1. Preface
  2. Understanding Pipeline Partitioning
  3. Partition Points
  4. Partition Types
  5. Pushdown Optimization
  6. Pushdown Optimization and Transformations
  7. Real-time Processing
  8. Commit Points
  9. Row Error Logging
  10. Workflow Recovery
  11. Stopping and Aborting
  12. Concurrent Workflows
  13. Grid Processing
  14. Load Balancer
  15. Workflow Variables
  16. Parameters and Variables in Sessions
  17. Parameter Files
  18. FastExport
  19. External Loading
  20. FTP
  21. Session Caches
  22. Incremental Aggregation
  23. Session Log Interface
  24. Understanding Buffer Memory
  25. High Precision Data

Advanced Workflow Guide

Advanced Workflow Guide

Configuring a Mapping for Recovery

Configuring a Mapping for Recovery

You can configure a mapping to enable transformations in the session to produce the same data between the session and recovery run. When a mapping contains a transformation that never produces repeatable data, you can add a transformation that always produces repeatable data immediately after it.
For example, you connect a transformation that never produces repeatable data directly to a transformation that produces repeatable data based on input order. You cannot configure recovery to resume from the last checkpoint unless the data is repeatable. To enable the session for recovery, you can add a transformation that always produces repeatable data after the transformation that never produces repeatable data.
The following figure shows a mapping that you cannot recover with resume from the last checkpoint:
This mapping includes two Source Qualifier transformations that connect to a Union transformation.
The mapping contains two Source Qualifier transformations that produce repeatable data. The mapping contains a Union and Custom transformation that never produce repeatable data. The Lookup transformation produces repeatable data when it receives repeatable data. Therefore, the target does not receive repeatable data and you cannot configure the session to resume recovery.
You can modify the mapping to enable resume recovery. Add a Sorter transformation configured for distinct output rows immediately after the transformations that never output repeatable data. Add the Sorter transformation after the Custom transformation.
The following figure shows the mapping with a Sorter transformation connected to the Custom transformation:
This mapping includes two Source Qualifier transformations that connect to a Union transformation, which connects to a Custom transformation, which connects to a Sorter transformation.
The Lookup transformation produces repeatable data because it receives repeatable data from the Sorter transformation.
The following table describes when transformations produce repeatable data:
Transformation
Repeatable Data
Aggregator
Always.
Application Source Qualifier
Based on input order.
Custom
Based on input order. Configure the property according to the transformation procedure behavior.
Data Masking
Based on input order. Configure the property according to the transformation procedure behavior. To produce repeatable data, configure repeatable masking or key masking for each port.
Expression
Based on input order.
External Procedure
Based on input order. Configure the property according to the transformation procedure behavior.
Filter
Based on input order.
HTTP
Based on input order. Configure the property according to the transformation procedure behavior.
Joiner
Based on input order.
Java
Based on input order. Configure the property according to the transformation procedure behavior.
Lookup, dynamic
Always. The lookup source must be the same as a target in the session.
Lookup, static
Based on input order.
MQ Source Qualifier
Always.
Normalizer, pipeline
Based on input order.
Normalizer, VSAM
Always. The normalizer generates source data in the form of unique primary keys. When you resume a session the session might generate different key values than if it completed successfully.
Rank
Always.
Router
Based on input order.
Sequence Generator
Always. The Integration Service stores the current value to the repository.
Sorter, configured for distinct output rows
Always.
Sorter, not configured for distinct output rows
Based on input order.
Source Qualifier, flat file
Always.
Source Qualifier, relational
Based on input order. Configure the transformation according to the source data. The Integration Service stages the data if the data is not repeatable.
SQL Transformation
Based on input order. Configure the transformation according to the source data.
Stored Procedure
Based on input order. Configure the property according to the transformation procedure behavior.
Transaction Control
Based on input order.
Union
Never.
Unstructured Data
Based on input order. Configure the property according to the transformation procedure behavior.
Update Strategy
Based on input order.
XML Generator
Always.
XML Parser
Based on input order. Configure the transformation according to the source data.
XML Source Qualifier
Always.
You can configure the Output is Repeatable and Output is Deterministic properties for the following transformations, or you can add a transformation that produces repeatable data immediately after these transformations:
  • Application Source Qualifier
  • Custom
  • External Procedure
  • Source Qualifier, relational
  • Stored Procedure

0 COMMENTS

We’d like to hear from you!