Metadata Manager extracts different metadata from the data flow based on the types of transformations that are present in the data flow.
Metadata Manager applies different metadata extraction rules when the following transformations are present in the data flow:
Audit transformation
This transformation adds columns to the data flow. Metadata Manager displays columns that originate from the source but not columns that are added by transformations in the data flow. Therefore, Metadata Manager does not update the target object with any column that the Audit transformation adds to the data flow.
Aggregate, Character Map, Data Conversion, Derived Column transformations
These transformations apply function expressions to columns values, for example,
AVG(SalesAmount)
or
UPPERCASE(LastName)
. Metadata Manager displays the function expressions in the
Operation
property in the target column. If multiple expressions apply to a target column, Metadata Manager concatenates the expressions.
Copy Column transformation
This transformation copies input columns and creates new columns in the transformation output. Because the copied columns originate from the source, Metadata Manager displays the new columns in the target.
Export Column transformation
This transformation inserts data into one or more flat files. Because the flat files originate from the transformation, Metadata Manager does not display them in the metadata catalog or in the data lineage diagram.
Fuzzy Lookup, Lookup, Term Lookup transformations
Metadata Manager displays the reference data set as a source object. Metadata Manager displays only the return columns in the reference data set.
Pivot transformation
In the Pivot transformation, each input column has a unique lineage ID and each output column has a source column. Metadata Manager creates a lineage link when the lineage ID value in an input column matches the source column value in an output column.
Script component
The Script component can be used as a source, target, or transformation. Metadata Manager groups all Script components under a connection called “_Scripts_” in the metadata catalog. Metadata Manager displays each Script component input and output group as a data set. It also displays each input and output column as a data attribute.
When the Script component is used as a source or target, Metadata Manager displays it in the data lineage diagram as a source or target object. When the Script component is used as a transformation, Metadata Manager uses the
Hide transformation scripts
configuration property to determine how to display the transformation logic.
Metadata Manager displays the transformation logic in the following ways based on the value of the
Hide transformation scripts
configuration property:
If the property is true, Metadata Manager does not display the transformation inputs or outputs. The data lineage diagram shows links between all columns in the source object and all columns in the target object.
If the property is false, Metadata Manager creates a separate object in the lineage diagram to represent each input or output group of a transformation script. The data lineage diagram displays links between the source object and the transformation script inputs and between the transformation script outputs and the target. However, it does not display links between the transformation script inputs and outputs. To define the flow of metadata within a transformation script, use rule-based links or enumerated links to link data attributes to each other. You can also use rule-based links or enumerated links to link transformation scripts to other metadata objects.