To provide for flexible recovery from both planned and unplanned exceptions that disrupt replication, Data Replication records checkpoint information separately for the Extractor, Server Manager, and Applier. By recording checkpoint information for each of these components, Data Replication can prevent data loss and ensure data consistency across all of the replication stages.
Data Replication stores checkpoint information in SQLite databases. Data Replication creates a separate SQLite database for the Extractor, Server Manager, and Applier task, where the task runs. These SQLite databases are in addition to the SQLite databases that Data Replication uses to store configuration file and internal information for the Extractor, Applier, and InitialSync processing.
SQLite is installed as part of the Data Replication installation. Data Replication creates and maintains all of its SQLite databases.
The following events trigger Extractor checkpoints:
The Extractor reaches the end of a log file, which can be an Oracle archive or online redo log, a MySQL binary log,
or a SQL Server backup file
.
The Extractor reads all available records from the source database logs.
In Continuous mode, the Extractor ends a microcycle of the duration that is specified in the
Continuous Replication Latency
option.
For Microsoft SQL Server sources, the Extractor reads a chunk of a log file. The chunk size is specified by the
extract.mssql.checkpoint_size
parameter.
For DB2 for Linux, UNIX, and Windows sources, the Extractor reads a chunk of a log file. The chunk size is specified by the
extract.db2.checkpoint_size
parameter.
For Oracle sources, the Extractor tries to read an online redo log that the Oracle database overwrote with new data.
For Oracle source instances in a RAC environment, an Oracle instance stops and the Extractor processes all of the redo logs in the thread.
The following events trigger Applier checkpoints:
The Applier commits data to the target.
The Applier uses subtask threads to apply changes, and a thread that applies a primary key update writes a commit to the target.
For Apache Kafka targets, the Applier saves the sequence of the last change operation successfully sent to the target as a "checkpoint" to a checkpoint file, provided that you use the default guaranteed delivery mode. If you do not use guaranteed delivery, the Applier writes a checkpoint after each Commit operation. The checkpoint file must exist on the system where the Applier, the CDC Publisher, and a Server Manager instance (Main server or subserver) run. You can change the checkpoint file name or directory by editing the apply.kafka.kafka_checkpoint_file_name and apply.kafka.kafka_checkpoint_file_directory runtime parameters. By default, the checkpoint file name matches the configuration name.