Table of Contents

Search

  1. Preface
  2. Transformations
  3. Source transformation
  4. Target transformation
  5. Aggregator transformation
  6. Cleanse transformation
  7. Data Masking transformation
  8. Deduplicate transformation
  9. Expression transformation
  10. Filter transformation
  11. Hierarchy Builder transformation
  12. Hierarchy Parser transformation
  13. Hierarchy Processor transformation
  14. Input transformation
  15. Java transformation
  16. Java transformation API reference
  17. Joiner transformation
  18. Labeler transformation
  19. Lookup transformation
  20. Mapplet transformation
  21. Normalizer transformation
  22. Output transformation
  23. Parse transformation
  24. Python transformation
  25. Rank transformation
  26. Router transformation
  27. Rule Specification transformation
  28. Sequence Generator transformation
  29. Sorter transformation
  30. SQL transformation
  31. Structure Parser transformation
  32. Transaction Control transformation
  33. Union transformation
  34. Velocity transformation
  35. Verifier transformation
  36. Web Services transformation

Transformations

Transformations

Deduplication and consolidation operations

Deduplication and consolidation operations

When you run a mapping with a Deduplicate transformation, the transformation analyzes the identity data in each input record. The transformation generates a set of percentage scores that represent the degrees of similarity between the input records. If two or more records match one another with scores that exceed a given threshold, the transformation considers the records to be duplicates.
The deduplicate asset that you add to the transformation specifies the comparison criteria for the deduplication operation, including the threshold score that duplicate records must meet.
Consolidation is an optional process that the deduplicate asset can specify for the transformation. During consolidation, the transformation evaluates the sets of matching records that the deduplication process identifies. The transformation selects or constructs a preferred version of the records in each set.
A
Data Quality
user configures the deduplication and consolidation processes in the deduplicate asset. For more information about the criteria that the asset defines, contact the
Data Quality
user.

Rules and guidelines for deduplication and consolidation

When you add a Deduplicate transformation to a mapping, consider the following rules and guidelines:
Mapping fields for identity analysis
The deduplicate asset that you add to the transformation specifies a type of identity, such as a person name or an organization name. The asset identifies the identity type as the
objective
of the deduplication operations. The type of identity on the asset defines the types of information that the transformation expects to find in the input fields.
You must map the appropriate input fields on the transformation to the target fields that the transformation indicates. You can optionally map the optional input fields to other fields.
Scores and thresholds
The Deduplicate transformation calculates a score for each possible pair of records in the input data. The transformation returns the scores for the records within each set of matching duplicate records. It does not return the scores for records that do not belong in the same set.
The transformation represents the relationships between the records in a matching set as a link score and a driver score.
Sequence ID and group key fields
On the
Field Mapping
tab, the transformation adds a group key field and a sequence ID field to the fields that the asset specifies. The group key field is mandatory, whereas the sequence ID field is optional.
The group key is a data value that allows the transformation to sort the input records into subsets and to perform discrete duplicate analyses on each subset. When you select a suitable group key, you reduce the time that the mapping takes to run without reducing the quality of the mapping results. If you do not want to divide the input records into groups, add a field to the input data that contains a single or constant value and select the field as the group key.
The sequence ID values determine the order in which the transformation reads the input records. If your input records do not contain a sequence ID field, the transformation reads the records in the order in which they appear in the input data set.
Metadata fields
On the
Output Fields
tab, the transformation adds fields that display the score values for pairs of matching records. The fields also identify the set of matching records to which each record belongs. If the deduplicate asset specifies a consolidation process, the metadata fields specify a preferred record for each record set. The transformation identifies the preferred record as the
survivor
record.
Use the fields to understand the mapping results.