Operational Insights

Back Next

Database ingestion and replication job details

Database ingestion and replication
job details

You can drill down on a job to see job details from the

My Jobs

page accessed from the

Data Integration

Home page, from the

All Jobs

and

Running Jobs

pages in

Monitor

, and from the

Overview

and

All Jobs

tabs on the

Data Ingestion and Replication

page in

Operational Insights

After you click a job name, job information is displayed in the

Task Summary

Overview

Object Detail

, and

Alerts

panes. The

Alerts

pane is displayed only for incremental load and combined initial and incremental load jobs in Operational Insights.

The following image displays the panes:

Clicking the expander arrow next to the pane name expands or collapses the

Task Summary

Object Detail

, and

Alerts

panes.

Task Summary

View a diagram that shows the source and target connector types and connection names. It also shows the calculated data throughput, in rows per second if the job has successfully replicated data to the target, regardless of the job's current status. If the calculated value is 0, indicating no data has flowed to the target, the throughput is not displayed.

In the upper right corner
, the

Status

field displays the job status, which can be one of the following values:

Up and Running

. The job is running.

Running with Warning

. The job is running with a warning. This state can also occur when one or more table-specific subtasks fail but some subtasks are still running.

On Hold

. The job is in a paused state while the

Database Ingestion and Replication

(DBMI) agent is being updated.

Stopping

. The job is stopping in response to a Stop request.

Stopped

. The job was intentionally stopped.

Failed

. The job ended abnormally, the task deployment to the job failed, or one or more table-specific subtasks failed. Also, for an initial load job, the job was stopped.

Deploying

. The job is being deployed.

Deployed

. The job has been deployed.

Aborting

. The job is stopping immediately in response to an Abort request.

Aborted

. The job has been aborted.

Undeploying

. The job is being undeployed.

Undeployed

. The job has been undeployed.

Completed

. The job completed successfully.

Also, for incremental load jobs and combined initial and incremental load jobs, you can download the job execution log for the entire job run. In the Select log
list under the diagram, select one of the following log types:

Complete Log

. The entire log, including all types of messages. It is available for any job that ran, regardless of its state.

Error

. The error log, which includes messages only for errors that occurred. It is available for Failed jobs only. Use this log to determine the reason for the job failure, for example, the deployment failed. If the log file ends with an ellipsis (...), the log has been truncated because of its long length. In this case, download the Complete Log to see all error messages.

Overview

View information about the job, including the runtime environment, task name, load type, task type, task definition location, user who started the job, records read, records written, number of subtasks, and schedule name, if a schedule was used.

The circle image displays the number of subtasks on source tables by status. The color of the circle's rim corresponds to the status.

The following table describes the summary job properties and statistics:

Property	Description
Duration (on left)	The amount of time, in the hh:mm:ss format, that the job ran before it ended.
Start Time (on left)	The date and time when the job was deployed.
End Time (on left)	The date and time when the job ended because it completed processing, was stopped, or failed. This field is not displayed for running jobs
Runtime Environment	The name of the runtime environment that the job uses to run.
Agent Name	For incremental load and combined initial and incremental load jobs, the name of the Secure Agent that the job uses to run. In case of a failover from one Secure Agent to another, Database Ingestion and Replication displays the latest Secure Agent assigned to the job. For initial load jobs, the Agent Name is displayed in the Object Detail pane. For combined initial and incremental load jobs, the Agent Name is displayed both in the Task Summary and the Object Detail panes.
Task Name	The name of the associated database ingestion and replication . You can click the task-name link to view or edit task details in Data Ingestion and Replication , if necessary. If you edit the task, you must redeploy it for the updated task definition to be used for a job.
Load Type	The type of load operation that the job performs. Options are: Initial Load . Loads a snapshot of source data read at a specific point-in-time to a target. Incremental Load . Loads incremental data changes to a target on a continuous basis, until the job is stopped or ends. Initial and Incremental Load . Performs an initial load and then automatically switches to an incremental load.
Task Type	The type of task, which is Database Ingestion and ReplicationTask .
Task Location	The project or project folder that contains the database ingestion and replication task definition.
Started By	The name of the user who started the job.
Records Read	The number of records that were read from the source. For a combined initial and incremental load job, the Records Read count might be greater than the total number of object-level DML change records written. This behavior occurs because the initial load or resync processing always starts after change data capture has begun. As a result, some change records are included in the Records Read count and then discarded before initial load or resync processing starts. These discarded records cause the Records Write count to be less than the Records Read count.
Records Written	The number of records that were successfully replicated to the target. The Records Written value might be different from the Records Read value if source records are discarded. For example, in a combined initial and incremental load job, change records read from the source before the initial unload phase completes are discarded because they're not yet needed.
Subtasks	The number of subtasks that the database ingestion and replication job used to replicate data from source tables to the target. When a job runs, it uses a separate subtask to process each source table.
Capture Progress	For incremental load and combined initial and increment load jobs, the date and time in the source change stream to which capture processing has progressed, as shown in the time zone of the user profile.
Schedule	For initial load jobs, the name of the schedule that is used to run the job or "No schedule" if you run the job manually.

Object Detail

On the

Object Detail

pane lists subtasks on the source tables. You can view statistics and status information by source table from the last run of a

database ingestion and replication

job. When you click the expander arrow next to an object name, counts of processed inserts, updates, deletes, and LOB changes are shown for the table.

The following image shows a sample expanded object row:

This pane shows information for the latest job run. This tab is blank for jobs that have not run or are resuming.

The following table describes the

Object Detail

fields that are displayed for each table, depending on the load type and status:

Column	Description
Object	The name of the source table or view for which data was propagated to the target. For an incremental load job or a combined initial and incremental load job, click the arrow icon to the left of the object name to display detailed counts of Inserts, Deletes, Updates, LOBs, and DDL statements processed. For a combined initial and incremental load job, the Unload Count field is also displayed to show the number of records that the initial load portion of processing read from the source. The following usage notes apply to the detailed CDC counts: The counts are only for the current job run. If you stop and restart the job, the counts start over from zero. Do not use these counts to identify the number of rows written to the target. The counts are based on rows read from the source and do not reflect the records written to the target. Target write operations might be optimized by combining operations and reducing the number of physical writes. In this case, the counts might not match the number of write operations. The value N/A means that the count value is not applicable for the count type or the value has not yet been calculated. The Unload Count might not reflect the number of source records at the time the job is started or resynchronized because of a delay in the start of unload processing. Between the time of the unload request and start of unload processing, rows might be added to or deleted from the source table.
Target Object	The name of the target object that is mapped to the source object.
Agent Name	For initial load and combined initial and incremental load jobs, the name of the Secure Agent on which the job was run. For incremental load jobs, the Agent Name is displayed in the Overview section of the Task Summary pane. For combined initial and incremental load jobs, the Agent Name is displayed both in the Task Summary and the Object Detail panes.
Records Read	For an initial load job, the number of records that were read from the source. For other load types, this information is available only at the job-level on the Job Overview tab.
Records Written	For an initial load job, the number of records that were successfully written to the target. For other load types, this information is available only at the job-level on the Job Overview tab. The Records Read value can be greater than the Records Written value if some records read from the source were discarded. For example, in a combined initial and incremental job, any source change records read before the initial unload phase of the job has completed are discarded.
Task Duration	For an initial load job, the amount of time the subtask that processed the source table ran before it completed or was stopped. For other load types, this information is available only at the job-level on the Job Overview tab. When a job runs, it uses a separate subtask to process each source table.
Stage	For a combined initial and incremental load job, this column shows the stage in the transition from initial load processing to CDC processing for the table-specific job subtask. This column does not appear for other load types. The stage can be one of the following values: Not Started . Initial load processing has not yet started for the table, or if an error occurred and the table is in the Error on Retry state, the next attempt to process the table has not yet started. Started . Initial load processing has started. Unloading . The subtask is unloading data from the table as part of initial load processing. Unloaded . The subtask has finished unloading data from the table as part of initial load processing. Completed . The subtask completed initial load processing of the table. Normal . The subtask completed initial load processing of the table and has started CDC processing of the table. Cancelled . Initial load processing was cancelled or stopped. Error . The subtask detected an error in the source table. Actions menu > Resync For a subtask in a combined initial and incremental load job, if the subtask stage is Normal and the subtask status is any status other than Queued or Starting , the Actions menu is displayed on the right end of the subtask row. From the Actions menu, you can select Resync to resynchronize the source and target objects. For more information, see "Resynchronizing source and target objects" in Data Ingestion and Replication help.
Status	The status of the job subtask for the source object. If the job stops running, the subtask status reflects the status last collected before the job ended. For example, the job might be aborted but the subtask is in a Running status. The state can be one of the following values: Queued . The subtask has not yet started running. Starting . The subtask is starting. Started . For a combined initial and incremental load job, the subtask has started. Running . The subtask is running. On Hold . The subtask, as well as the job, is in a paused state while the Database Ingestion and Replication (DBMI) agent is being updated. Completed . The subtask completed processing successfully. Stopping . The subtask is stopping in response to a Stop request. Stopped . The subtask has stopped. Aborting . The subtask is ending immediately in response to an Abort request. Aborted . The subtask has been aborted. Failed . The subtask ended unexpectedly. Error . The subtask is in error and no longer writing data to the target table. For a combined initial and incremental load job, the subtask might be running and processing incremental change data but no data is being sent to the target. Error on Retry . An error occurred on the last retry of subtask processing, and now the subtask is waiting to retry processing again. If a DDL change occurs on a source table and then you resume the job, the table subtask state might not change as expected until the first DML operation occurs on the source table.
Data Validation	For initial load jobs that have an Oracle or a SQL Server source and a Snowflake target, if the task status is Completed , you can run data validation to compare the source and target data. This column shows the status of the data validation task. This column does not appear for other load types and other source and target combinations, or if the task is in any other status than Completed . From the Actions menu displayed on the right end of the subtask row, you can select Run Data Validation to start the validation task. For more information, see "Running data validation for a database ingestion and replication job" in Data Ingestion and Replication help. The data validation status can be one of the following values: N/A . The data validation has not been run. Processing . The data validation is running. Success . The data validation completed successfully. Click on the Success status to see the data validation summary. To download the data validation report, click the Download icon. Error . An error occurred during the data validation execution. To view the error message, click the Download icon. Report Pending . The data validation has been requested but the Data Validation service on the Secure Agent that runs the task is down. No Comparable Data . The source or target table does not contain any data.
CDV Last Run	If data validation is available for the task, this column shows the date and time when the data validation was last performed for the given subtask. If the data validation has not been run, the field value is N/A . This column appears in initial load jobs that have an Oracle or a SQL Server source and a Snowflake target, only if the task status is Completed .
Log	You can download a job execution log for a source object. The type and availability of the log depends on the load type and status. Options are: Complete . The complete log for an object subtask from job execution. This log type is available for a Completed, Failed, or Aborted subtask in an initial load job. Error . The log that contains error messages. This log type is available only for a Failed or Error subtask in an initial load or incremental load job. Stage Log . The log that covers the transition from the initial processing phase to the incremental processing phase in a combined initial and incremental load job for a source object. To download a log, click the Download icon. If you undeployed the job, you can download the log for a table only if the associated task has not been deleted. For incremental load jobs, you can get the complete log and error log for the entire job run from the Task Summary pane.

To control the line spacing in the list, click the

Settings

icon next to the

Find

box.

Alerts

The

Alerts

pane appears on the

Data Ingestion and Replication

page in

Operational Insights

for the selected incremental load or combined initial and incremental load job. On the

Alerts

pane, you can view alert messages that appear for certain events, such as source schema changes, during incremental load or combined initial and incremental load processing.

The following image shows this pane with an alert example:

The

Alerts

pane displays alert messages for all detected schema changes even if you set the schema drift options for the associated task to Ignore.

You can filter the list of alerts based on severity or a date range. To specify a date range, enter one of the following types of values in the

Filter

field:

Any Time

for all stored alerts.

Today

for alerts issued today from midnight to 11:59 pm.

Last Week

Last Month

, or

Last Year

to show alerts from the beginning of last week, month, or year to present.

Custom

to specify a custom date range that consists of a beginning date and time and an ending date and time.

The list of alerts includes the following columns:

Column	Description
Level	Severity level of the alert message, such as Critical or Warning.
Code	Alphanumeric code that identifies the alert type followed by the date and time when the event occurred.

Click the expander arrow to display a description of the event.

You can configure alert notifications for

database ingestion and replication

jobs from the

Alerts

Data Ingestion and Replication Alerts

page in

Operational Insights

then sends

Data Ingestion and Replication

alert notifications to the users and user groups you select, whenever a

database ingestion and replication

job acquires one of the configured statuses, detects a DDL change, or a delay in the change data capture processing occurs.

Rename Saved Search

Table of Contents

Operational Insights

Operational Insights

Database ingestion and replication job details

Database ingestion and replication job details

Task Summary

Overview

Object Detail

Alerts

Database ingestion and replication
job details