Amazon S3 Sources

Back Next

Configure data observability

Enable the data observability capability to receive anomaly notifications for metadata extracted from the source system.

To include profiling anomalies, enable data profiling for the catalog source. For accurate results, set the

Profiling Scope

option to

Full

in the

Data Profiling and Quality

configuration.

Click the

Data Observability

tab.

Select

Enable Data Observability

In the

Parameters

area, enter a value to specify the minimum number of catalog source job runs required to start detecting anomalies. For example, if you enter a value of 5, then the job starts to detect anomalies from the sixth run.

This option does not apply to anomalies that require a minimum of two runs to generate.

Specify the maximum number of anomaly events to generate for each catalog source run. Default is 1000.

Optionally, in the

Asset Filters

area, enable filters to run data observability on specific assets.

Specify data profiling filters if you want to override the data profiling filter and extract profiling anomalies for assets based on a different filter.

To add a data profiling filter, perform the following steps:

Select

Yes

to view filter options.

From the

Include/Exclude

list, choose to include or exclude metadata.

From the

Specify data profiling filters

list, select an object type.

Enter a value to specify the object location.

For example, you ran a profile on multiple schemas, and now you want to generate profiling anomalies on a specific schema. Enter the schema name in the input field.

To include or exclude multiple objects, click the

Add

icon to add filters with the OR condition.

Optionally, in the

Metric Filter

area, specify filters to generate events based on specific metrics. If you don't include a filter, the job detects both profiling and other anomalies applicable to the catalog source.

Select

Filter conditions

From the

Include Metric

list, choose to include or exclude a metric.

From the

All Metrics

list, select a metric for which you want to generate events.

From the

Sensitivity

list, select the sensitivity of the anomaly.

From the

Select detection rules

list, select rules that you apply to detect anomalies.

The following table describes the detection rules that you can select:

Option	Description
Static Data	Detects the following anomalies: Percentage variation. If percentage values other than 0% or 100% remain constant for three or more profile runs, any subsequent change generates a Percentage Variation anomaly. Count variation. A change in a value that is usually constant indicates that there is an anomaly.
100% or 0% Change Detection	Detects the following anomalies: Drop from maximum. A drop from 100% to a lower value generates a Drop from Maximum anomaly. Surge from minimum. A spike from 0% to any higher value generates a Surge from Minimum anomaly.
Standard Deviation	Detects the following anomalies: Drop in transition. A drop from a value that has been decreasing at a constant rate to a significantly lower value generates a Drop in Transition anomaly. Surge in transition. A spike from a value that has been increasing at a constant rate to a significantly higher value generates a Surge in Transition anomaly. Deviation. If the percentage values of a record change with each profile run, the algorithm calculates an expected range based on the changes observed in each profile run. A new value that falls outside of this expected range generates a Deviation anomaly.
Breaking Trends	Detect the following types of count-based anomalies: Drop. A drop in a value that has been increasing to a lower value generates a Drop anomaly. Surge. A spike in a value that has been decreasing to a higher value generates a Surge anomaly.

Rename Saved Search

Table of Contents

Amazon S3 Sources

Amazon S3 Sources

Configure data observability

Configure data observability