Google BigQuery Connectors

Back Next

Mapping tasks with CDC sources example

Your organization needs to replicate real-time changed data from a mission-critical production system to minimize intrusive, non-critical work, such as offline reporting or analytical operations system. You can use Google BigQuery V2 Connector to capture changed data from any CDC source and write the changed data to a Google BigQuery target. Add the CDC sources in mappings, and then run the associated mapping tasks to write the changed data to the target.

Data Integration

, click

New

Mapping

Create

The

New Mapping

dialog box appears.

Enter a name and description for the mapping.

On the Source transformation, specify a name and description in the general properties.

On the

Source

tab, select any configured CDC connection and specify the required source properties.

On the Target transformation, specify a name and description in the general properties.

On the

Target

tab, perform the following steps to configure the target properties:

In the

Connection

field, select the Google BigQuery V2 connection.

In the

Target Type

field, select the type of the target object.

In the

Object

field, select the required target object.

In the

Operation

field, select

Data Driven

to properly handle insert, update, and delete records from the source.

In the

Data Driven Condition

field, leave the field empty.

In the

Update Column

field, select the key columns to upsert or update data to or delete data from Google BigQuery.

In the

Advanced Properties

section, you must select CDC in the

Write Mode

property.

You can only configure the following advanced target properties for CDC mode:

Target Dataset ID

Target Table Name

Job Poll Interval In Seconds

pre SQL

post SQL

pre SQL Configuration

post SQL Configuration

On the

Field Mapping

tab, map the incoming fields to the target fields. You can manually map an incoming field to a target field or automatically map fields based on the field names.

In the

Actions

menu, click

New Mapping Task

The

New Mapping Task

page appears.

In the

Definition

tab, enter the task name and select the configured mapping.

In the

CDC Runtime

tab, specify the required properties for the selected CDC source.

For more information about the

CDC Runtime

properties, see the source properties for the selected CDC source.

On the

Runtime Options

tab, add the following properties in the

Advanced Session Properties

section:

Select

Commit on End of File

from the menu, and keep the property disabled.

Select

Recovery Strategy

and set

Resume from last checkpoint

as the value of the property.

Click

Save

Run

the mapping task.

Alternatively, you can create a schedule that runs the mapping task on a recurring basis without manual intervention. You can define the schedule to minimize the time between mapping task runs.

In Monitor
, you can monitor the status of the logs after you run the task.

Mapping tasks with CDC sources

Download Guide

Watch

Comments

Cloud Connectivity Homepage

Communities

Knowledge Base

Success Portal