Table of Contents

Search

  1. Preface
  2. Transformations
  3. Source transformation
  4. Target transformation
  5. Aggregator transformation
  6. Cleanse transformation
  7. Data Masking transformation
  8. Deduplicate transformation
  9. Expression transformation
  10. Filter transformation
  11. Hierarchy Builder transformation
  12. Hierarchy Parser transformation
  13. Hierarchy Processor transformation
  14. Input transformation
  15. Java transformation
  16. Java transformation API reference
  17. Joiner transformation
  18. Labeler transformation
  19. Lookup transformation
  20. Mapplet transformation
  21. Normalizer transformation
  22. Output transformation
  23. Parse transformation
  24. Python transformation
  25. Rank transformation
  26. Router transformation
  27. Rule Specification transformation
  28. Sequence Generator transformation
  29. Sorter transformation
  30. SQL transformation
  31. Structure Parser transformation
  32. Transaction Control transformation
  33. Union transformation
  34. Velocity transformation
  35. Verifier transformation
  36. Web Services transformation

Transformations

Transformations

Example: Add an ID column to nonpartitioned data

Example: Add an ID column to nonpartitioned data

Your organization runs a solar thermal power system that uses sensors to monitor the health of the system. Currently, each sensor is identified by its location. Instead, you want to identify each sensor using an ID to simplify future analytics on the data.
You collect the following data on sensor readings:
SensorLocation
LastReadingTime
Area A
7/9/2019 11:36:09
Area B
7/9/2019 16:43:42
Area C
7/9/2019 13:23:53
To add an ID column and assign ID values to each sensor, perform the following tasks:
Step 1. Create a Python transformation.
Create a Python transformation. On the
Advanced
tab, set the behavior to Passive.
Step 2. Pass data to the Python transformation.
Pass data from upstream transformations in the mapping to the Python transformation.
After you pass the data to the Python transformation, it contains the following incoming fields:
  • SensorLocation
  • LastReadingTime
Step 3. Create output fields.
Use the
Output Fields
tab in the Python transformation to create the output field
SensorID_out
to represent the ID column.
Additionally, create the following output fields to pass incoming data to downstream transformations:
  • SensorLocation_out
  • LastReadingTime_out
Step 4. Set the ID value for each row.
In the
Main Python Code
section, set the ID value for each row that is processed and write the data to the output fields using the following code:
SensorID="".join(str(x) for x in map(ord, SensorLocation)) SensorID_out = SensorID SensorLocation_out = SensorLocation LastReadingTime_out = LastReadingTime
Step 5. Run the mapping.
If the output fields in the Python transformation are linked directly to a Write transformation, the target contains the following data after you run the mapping:
SensorID_out
SensorLocation_out
LastReadingTime_out
65114101973265
Area A
7/9/2019 11:30:00
65114101973266
Area B
7/9/2019 11:35:00
65114101973267
Area C
7/9/2019 11:40:00