Table of Contents

Search

  1. Preface
  2. Transformations
  3. Source transformation
  4. Target transformation
  5. Aggregator transformation
  6. Cleanse transformation
  7. Data Masking transformation
  8. Data Services transformation
  9. Deduplicate transformation
  10. Expression transformation
  11. Filter transformation
  12. Hierarchy Builder transformation
  13. Hierarchy Parser transformation
  14. Hierarchy Processor transformation
  15. Input transformation
  16. Java transformation
  17. Java transformation API reference
  18. Joiner transformation
  19. Labeler transformation
  20. Lookup transformation
  21. Machine Learning transformation
  22. Mapplet transformation
  23. Normalizer transformation
  24. Output transformation
  25. Parse transformation
  26. Python transformation
  27. Rank transformation
  28. Router transformation
  29. Rule Specification transformation
  30. Sequence Generator transformation
  31. Sorter transformation
  32. SQL transformation
  33. Structure Parser transformation
  34. Transaction Control transformation
  35. Union transformation
  36. Velocity transformation
  37. Verifier transformation
  38. Web Services transformation

Transformations

Transformations

Example: Selecting a group key column

Example: Selecting a group key column

Let's say that a bank wants to search for duplicate bank account holders. The bank's customer data set includes columns for customer names and addresses, and the bank chooses
Contact
as the objective in the deduplicate asset. The bank decides to sort the input records into groups and to perform duplicate analysis on each group. The bank must select a column in the Deduplicate transformation on which to create the groups.
The following table shows a fragment of the data set:
Customer ID
Lastname
Firstname
Address1
City
State
Zip
Country
90999990
Armstrong
Al
6121 SUNSET BLVD.
LOS ANGELES
CA
90028
USA
90999907
Baldwin
Lynn
1600 EL CAMINO REAL, SUITE 1500
MENLO PARK
CA
94025
USA
90999917
Baldwyn
Linn
1600 EL CAMINO REAL, #1500
MENLO PK
CA
94025
USA
90999859
Belleperche
Carmen
9255 SUNSET BLVD.
LOS ANGELES
CA
90069
USA
90999876
Clark
Wick
777 S. FIGUEROA
LOS ANGELES
CA
90071
USA
90999859
Bachtin
Guy
30 S. WACKER
CHICAGO
IL
60606
USA
90999868
Dicintio
David
181 WEST MADISON ST
CHICAGO
IL
60602
USA
90999869
Ash
Pascal
335 WEST 16TH STREET
NEW YORK
NY
10011
USA
90999996
Bachtin
David
1633 BROADWAY
NEW YORK
NY
10022
USA
90999994
Carpenter
Brad
30 BROAD ST
NEW YORK
NY
42304
USA
90999820
Dedmond
David
ONE FINANCIAL SQUARE
NEW YORK
NY
10008
USA
90999902
Backwell
Chris
901 SE OAK, WILLAMETTE PLZ
PORTLAND
OR
97214
USA
90999897
Askerup
Nancy
400 MARKET STREET
HOUSTON
TX
77027
USA
90999904
Choy
Shelley
1177 WEST LOOP SOUTH
HOUSTON
TX
77027
USA
90999886
Cote
Lian
530 E. SWEDESFORD RD.
HOUSTON
TX
77027
USA
90999999
Croteau
Paul
3829-55 GASKINS ROAD
HOUSTON
TX
77027
USA
In this scenario, you might identify the State column as the most suitable column on which to sort the records. You select the State column as the
GroupKey
field in the transformation.
When you select the State column as the GroupKey field, the deduplication operation enables the creation of five groups, one for each state. The likelihood that the bank has customers with the same contact information in different states is very low. Additionally, the data includes a Customer ID column that can add to the confidence of the deduplication process.
The Customer ID column is a poor candidate for group creation, as it is a primary key field. If you select the column as the GroupKey field, the deduplication operation creates a group for every unique ID and thus for every record.
The Country column is also a poor candidate for group creation, as the column contains the same value in every row. If you select the Country column as the GroupKey field, the deduplication operation adds all of the records to the same group. Your bank might have two or more genuine customers with the same name living across the country, and you do not want to deduplicate their entries.

0 COMMENTS

We’d like to hear from you!