Amazon S3 Sources

Back Next

Configure metadata extraction

When you configure the

Amazon S3

catalog source, you choose a runtime environment, define filters, and enter configuration parameters for metadata extraction.

In the

Connection and Runtime

area, choose a serverless runtime environment or the Secure Agent group where you want to run catalog source jobs.

Serverless runtime environment options are available if the catalog source works with a serverless runtime environment.

Choose to retain, delete, or deprecate objects that are deleted from the source system in the catalog with the

Metadata Change Option

Retain

. Retains objects that are deleted from the source system in the catalog. If you update or add a filter, the catalog retains objects extracted from the previous job and extracts additional objects that match the current filter. Objects deleted from the source system are not deleted from the catalog. Enrichments added on deleted objects and relationships are retained.

Delete

. Deletes metadata from the catalog based on objects deleted from the source system and changes you make to the filter. Enrichments added on deleted objects and relationships are also permanently lost. Objects renamed in the source system are removed and recreated in the catalog.

Deprecate

. When objects are deleted from the source system, the catalog changes the lifecycle of the imported objects to Obsolete. The lifecycle of objects also changes to Obsolete when you add exclusion filters or reduce the scope so that objects extracted in earlier jobs are no longer included. For example, the first job includes the sales schema. The second job also includes the sales schema, but excludes the sales.transactions table. In this case, the second job marks the sales.transactions object as Obsolete, even though the first job includes it. This does not impact enrichments applied to deprecated objects and relationships. Objects renamed in the source system are removed and recreated in the catalog. When you run the catalog source job again for other capabilities such as data classification, relationship discovery, or glossary association, the job doesn't consider obsolete objects. Obsolete objects remain in the catalog until they are purged when you run a

Purge Obsolete Objects

job on the

Explore

page.

You can also change the configured metadata change option when you run a catalog source.

In the

Filters

area, define one or more filter conditions to apply for metadata extraction.

To define filters, you can either select an object type and enter the path to the object as the filter value, or select an object from a list of objects available in the source system.

Select

Yes

to view filter options.

From the Include/Exclude list, choose to include or exclude metadata based on the filter parameters.

Perform one of the following steps:

From the Object type list, select an object type, depending on the object that you want to extract metadata from. Enter the path to the object as the filter value.

In the filter value field, click the Search button and select an object from a list of objects available in the source system.

The Object type field updates based on the selected object.

If you select an object type and then click the Search button, the list of objects includes all object types, but you can only select objects that match the selected object type.

You can edit the filter value after you select an object from the list.

You can only search for object types that work with the search functionality. If you don't see the Search button for the selected object, enter the object path as the filter value.

If the object metadata is available in

Data Governance and Catalog

, a check mark appears next to the object.

To select an object, you need to have permissions on the connection to the source system.

The following image shows the filter condition options:

The filters contain options to include or exclude
metadata based on specific object types.

To define an additional filter with an OR condition, click the Add
icon.

In the

Configuration Parameters

area, enter the configuration parameters.

The following table describes the properties that you can enter in the

Catalog Source Configuration Options

section:

Property	Description
Extract Group Elements from Hierarchical Files	Select one of the following options to extract group or leaf elements from hierarchical files: Yes. Extracts group elements from hierarchical files with the complete hierarchy of hierarchical fields. You can view the hierarchy of hierarchical files in the Hierarchy tab of assets in Data Governance and Catalog . No. Extracts only leaf elements from hierarchical files without the complete hierarchy of hierarchical fields. You can extract group elements from hierarchical files for the following file types: AVRO. Extracts and groups hierarchical files and hierarchical fields. Parquet. Extracts and groups hierarchical files and hierarchical fields. JSON. Extracts and groups hierarchical files and hierarchical fields. XML. Extracts and groups elements and attributes. For XML file types, a maximum depth of 1000 elements is permitted within a single element in the hierarchy. XSD. Extracts and groups elements and attributes. If you modify the Extract Group Elements from Hierarchical Files field and run the catalog source again, the asset page doesn't display the hierarchical elements in the correct hierarchy groups. If you modify the property value, purge the catalog source before you run it again.
Enable Extension-Based File Type Detection	Select one of the following options to detect file types by file extensions or by parsing the file contents: Yes. Detects file types by file extensions. No. Parses the file contents to detect file types. You can detect file types by file extensions for the following file types: CSV TSV TXT XML
Use First Row as Header of Delimited Files	Select one of the following options to use the first row as the header or detect headers automatically for delimited files: Yes. Detects column headers based on the following rules: Duplicate headers get suffixed with ‘#’ followed by a number, for example, ABC#1, ABC#2. The detection is not case-sensitive. Empty column header values appear as UnknownColumn<position>, for example UnknownColumn2. The header row in the file is detected even if it has a different number of columns than the data rows. No. Detects headers automatically for delimited files.
Headers of Delimited Files	Specify values to determine headers of delimited files. Separate multiple values by commas. If any value from the list is found in the first row of the delimited file, then the first row is used as the header. This parameter appears only if you choose No for the Use First Row as Header of Delimited Files parameter.
Treat Files Without Extension As	Select one of the following options to use to identify files without an extension: Parquet Avro JSON
Enter File Delimiter	Specify the file delimiter if the file from which you extract metadata uses a delimiter other than the following list of delimiters: Comma(,) Horizontal tab ( \t) Semicolon (;) Colon (:) Pipe symbol (\|) Enclose the delimiter in single quotes, such as '$'. Use a comma to separate multiple delimiters. Adding a custom delimiter overrides the default list of delimiters. If you specify a delimiter, characters from the default list are not considered as delimiters.
Files to be excluded during partition discovery	Specify the regular expression of the files that you want to exclude during partition discovery. Use a comma to separate multiple regular expressions. For example, enter ".json","Customer.csv","Parquet.".
Treat Unrecognized Files As	Choose how to handle files with unrecognized formats during metadata extraction. Select one of the following options: Structured files. Treats all files with unrecognized formats as structured files. Unstructured files. Treats all files with unrecognized formats as unstructured files. On a serverless runtime environment, unrecognized files are treated as structured files.
Unrecognized Files Override	Enter one or more file paths as exceptions to override whether the specific unrecognized files should be treated as structured or unstructured files. You can use the path-based overrides to apply different treatment rules to different files within the same catalog source.

The following table describes the properties that you can enter in the

Partitioned File Configuration

section:

Property	Description
Custom Partition Detection Configuration File	The configuration file in JSON format to detect custom partitions in the source system. The configuration file defines the pattern of the non-Hive style custom partitions. The system automatically detects partitions if the date format of the partition key is in any of the following formats: dd-MM-yy dd-MM-yyyy dd-MMM-yyyy For example, the system interprets dd-MM-yyyy as the partition format and extracts Customer as a hierarchical file instead of a folder in the following partition pattern: Customer \|__01-01-2022 \| \| \|__ dataFile1.parquet \| \| \|__ dataFile2.parquet \| \| \|__ ... \|__02-02-2022 \|__... The system doesn't automatically detect partitions with other patterns, such as MM-dd-yyyy. If the pattern is different, you need to define it in JSON format. For example, {"CustomPartitionPatterns": ["MM-dd-yyyy"]} To detect the epoch time format, define it in JSON format as: {"CustomPartitionPatterns": ["@"]}
Partitioned Pruning Configuration Options	The configuration file in JSON format for partition pruning. The configuration file contains additional information to identify partitions and determine the relevant schema in the source system.
Enable Partition Detection	Specify if you want to enable partition detection.
Enable Partition Pruning	Specify if you want to enable partition pruning. Default is Yes. Applicable if you enable partition detection. If you run the catalog source job to extract incremental metadata with partition pruning, the job considers the pruned partitioned data that is changed or new since the last job run.

Configure additional capabilities for the catalog source by clicking on the tabs.

Rename Saved Search

Table of Contents

Amazon S3 Sources

Amazon S3 Sources

Configure metadata extraction

Configure metadata extraction