Configure data profiling and quality

Click the

Data Profiling and Quality

tab.

Expand

Data Profiling

and select

Enable Data Profiling

.

Ensure that you have permissions on all the staging connections that you use in your data profiling configuration. You can't run the job if you don't have permissions on the connections that you use. Select connections that you have access to, or ask the administrator to grant the necessary permissions on the connections that you want to use.

In the

Connection and Runtime

area, choose the Secure Agent group where you want to run catalog source jobs.

Optionally, in the

Filters

area, specify additional filters in addition to metadata filters:

Select

Yes

to view filter options.

From the Include/Exclude list, choose to include or exclude metadata based on the filter parameters.

From the Object type list, select

File

or

Folder

.

Enter the path to the object as the filter value.

Filters can contain the following wildcards:

Question mark. Represents a single character.

Asterisk. Represents multiple characters.

Examples:

File: You extracted metadata from a folder that contains multiple files and now you want to run a profile on a specific file. Select

File

from the Object type option and select

Named

from the Filter Criteria option, and then enter the file name in the input field.

Folder: You extracted metadata from a folder that contains multiple folders and now you want to run a profile on a specific folder. Select

Folder

from the Object type option and select

Named

from the Filter Criteria option, and then enter the folder name in the input field.

To include or exclude multiple objects, click the

Add

icon to add filters with the AND condition.

In the

Parameters

area, configure the parameters.

The following table describes the parameters that you can enter:

Parameter	Description
Modes of Run	Determine the type of data that you want the data profiling task to collect. Choose one of the following options: Keep signatures only. Collects only aggregate information such as data types, average, standard deviation, and patterns. Keep signatures and values. Collects both signatures and data values.
Profiling Scope	Determine whether you want to run data profiling only on the changes made to the source system or on the entire source system. Choose one of the following options: Incremental. Includes only source metadata that is changed or updated since the last profile run. Full. Includes the entire metadata that is extracted based on the filters applied for extraction.
Sampling Type	You can run the data profiling task on all rows for an Amazon S3 source system.
Elastic Runtime Environment	Select an elastic runtime environment for complex file types, including Avro and Parquet.
Staging Connection	The staging connection where data profiling results are stored temporarily during the execution. Applicable for data profiling on complex file types.
Maximum Precision of String Fields	The maximum precision value for profiles on string data type.
Text Qualifier	The character that defines string boundaries. If you select a quote character, profiling ignores delimiters within the quotes. Select a qualifier from the list. Default is Double Quote.
Code Page for Delimited Files	Select a code page that the Secure Agent can use to read and write data. Use this option to ensure that profile results for assets with non-English characters don't include junk characters. Default value is UTF-8. Choose one of the following options: MS Windows Latin 1. Select for ISO 8859-1 Western European characters. UTF-8. Select for Unicode and non-Unicode characters. Shift-JIS. Select for double-byte characters. ISO 8859-15 Latin 9 (Western European). ISO 8859-2 Eastern European. ISO 8859-3 Southeast European. ISO 8859-5 Cyrillic. ISO 8859-9 Latin 5 (Turkish). IBM EBCDIC International Latin-1.
Escape character for delimited files	You can specify an escape character if you need to override the default escape character. An escape character ignores a delimiter character in an unquoted string if the delimiter is part of the string value. If you specify an escape character, the data profiling task overrides the default escape character that the Metadata Extraction job detects and considers the specified escape character. It then reads the delimiter character as a part of the string value. If you don't specify an escape character, the data profiling task considers the default escape character that the Metadata Extraction job detects and reads the delimiter character as a part of the string value.
Read Multiple Line JSON Files	By default, the data profiling job reads each JSON schema as a single line. Select Yes to read input that spans across multiple lines.

Expand

Data Quality

and select

Enable Data Quality

.

You can click

Use Data Profiling Parameters

to use the same parameters as in the

Data Profiling

section.

Ensure that you have permissions on all the staging and flat file connections that you use in your data quality configuration. You can't run the job if you don't have permissions on the connections that you use. Select connections that you have access to, or ask the administrator to grant the necessary permissions on the connections that you want to use.

In the

Connection and Runtime

area, choose the Secure Agent group where you want to run catalog source jobs.

In the

Parameters

area, configure the parameters.

The following table describes the properties that you can enter:

Parameter	Description
Data Quality Rule Automation	Enable the option to automatically create or update rule occurrences for data elements in the catalog source. Choose one of the following options: Apply on Data Elements linked with Business Dataset . Creates rule occurrences for all data elements that are linked with business data sets in the catalog source. Apply on all Data Elements . Creates rule occurrences for all data elements in the catalog source.
Data Quality Remediation	Enable the option to specify a flat file connection to store the list of failed rows so that users can remediate poor data quality scores. Choose one of the following options: No . Doesn't enable the data quality failure ticket option. Yes . Shows a list of flat file connections where you write failed rows to customer-managed locations.
Data Quality Failure Ticket	Specify whether you want to create data quality failure tickets for poor data quality scores based on the threshold defined for the rule occurrence in Data Governance and Catalog . Choose one of the following options: No . Doesn't automatically create data quality failure tickets when the data quality scores are poor. Yes . Automatically creates data quality failure tickets based on the data quality threshold values you define in Data Governance and Catalog , and notifies you when a data quality score is below the threshold. You must configure a workflow event for the data quality failure and enable the event in Metadata Command Center .
Cache Result	Specify how you want to preview rule occurrence results. Select Agent Cache if you want to generate a cache file in the runtime environment and to preview the cached results faster in subsequent data preview runs. The results are cached for seven days by default after the first run in the runtime environment. Select No Cache if you don't want to cache the preview results and view the live results.
Run Rule Occurrence Frequency	Specify whether you want to run data quality rules based on the frequency defined for the rule occurrence in Data Governance and Catalog.
Sampling Type	You can run the data quality task on all rows for an Amazon S3 source system.
Elastic Runtime Environment	Select an elastic runtime environment for complex file types, including Avro and Parquet.
Staging Connection	The staging connection where data quality results are stored temporarily during the execution. Applicable for data profiling on complex file types.
Maximum Precision of String Fields	The maximum precision value for profiles on string data type.
Text Qualifier	The character that defines string boundaries. If you select a quote character, profiling ignores delimiters within the quotes. Select a qualifier from the list. Default is Double Quote.
Code page for delimited files	Select a code page that the Secure Agent can use to read and write data. Use this option to ensure that rule results for assets with non-English characters don't include junk characters. Default value is UTF-8. Choose one of the following options: MS Windows Latin 1. Select for ISO 8859-1 Western European characters. UTF-8. Select for Unicode and non-Unicode characters. Shift-JIS. Select for double-byte characters. ISO 8859-15 Latin 9 (Western European). ISO 8859-2 Eastern European. ISO 8859-3 Southeast European. ISO 8859-5 Cyrillic. ISO 8859-9 Latin 5 (Turkish). IBM EBCDIC International Latin-1.
Escape character for delimited files	You can specify an escape character if you need to override the default escape character. An escape character ignores a delimiter character in an unquoted string if the delimiter is part of the string value. If you specify an escape character, the data quality task overrides the default escape character that the Metadata Extraction job detects and considers the specified escape character. It then reads the delimiter character as a part of the string value. If you don't specify an escape character, the task considers the default escape character that the Metadata Extraction job detects and reads the delimiter character as a part of the string value.
Read Multiple Line JSON Files	By default, the data quality job reads each JSON schema as a single line. Select Yes to read input that spans across multiple lines.

Table of Contents

Amazon S3 Sources

Amazon S3 Sources