Table of Contents

Search

  1. Preface
  2. Transformations
  3. Source transformation
  4. Target transformation
  5. Access Policy transformation
  6. Aggregator transformation
  7. Cleanse transformation
  8. Data Masking transformation
  9. Data Services transformation
  10. Deduplicate transformation
  11. Expression transformation
  12. Filter transformation
  13. Hierarchy Builder transformation
  14. Hierarchy Parser transformation
  15. Hierarchy Processor transformation
  16. Input transformation
  17. Java transformation
  18. Java transformation API reference
  19. Joiner transformation
  20. Labeler transformation
  21. Lookup transformation
  22. Machine Learning transformation
  23. Mapplet transformation
  24. Normalizer transformation
  25. Output transformation
  26. Parse transformation
  27. Python transformation
  28. Rank transformation
  29. Router transformation
  30. Rule Specification transformation
  31. Sequence Generator transformation
  32. Sorter transformation
  33. SQL transformation
  34. Structure Parser transformation
  35. Transaction Control transformation
  36. Union transformation
  37. Velocity transformation
  38. Verifier transformation
  39. Web Services transformation

Transformations

Transformations

Example: Add an ID column to nonpartitioned data

Example: Add an ID column to nonpartitioned data

Your organization runs a solar thermal power system that uses sensors to monitor the health of the system. Currently, each sensor is identified by its location. Instead, you want to identify each sensor using an ID to simplify future analytics on the data.
You collect the following data on sensor readings:
SensorLocation
LastReadingTime
Area A
7/9/2019 11:36:09
Area B
7/9/2019 16:43:42
Area C
7/9/2019 13:23:53
To add an ID column and assign ID values to each sensor, perform the following tasks:
Step 1. Create a Python transformation.
Create a Python transformation. On the
Advanced
tab, set the behavior to Passive.
Step 2. Pass data to the Python transformation.
Pass data from upstream transformations in the mapping to the Python transformation.
After you pass the data to the Python transformation, it contains the following incoming fields:
  • SensorLocation
  • LastReadingTime
Step 3. Create output fields.
Use the
Output Fields
tab in the Python transformation to create the output field
SensorID_out
to represent the ID column.
Additionally, create the following output fields to pass incoming data to downstream transformations:
  • SensorLocation_out
  • LastReadingTime_out
Step 4. Set the ID value for each row.
In the
Main Python Code
section, set the ID value for each row that is processed and write the data to the output fields using the following code:
SensorID="".join(str(x) for x in map(ord, SensorLocation)) SensorID_out = SensorID SensorLocation_out = SensorLocation LastReadingTime_out = LastReadingTime
Step 5. Run the mapping.
If the output fields in the Python transformation are linked directly to a Write transformation, the target contains the following data after you run the mapping:
SensorID_out
SensorLocation_out
LastReadingTime_out
65114101973265
Area A
7/9/2019 11:30:00
65114101973266
Area B
7/9/2019 11:35:00
65114101973267
Area C
7/9/2019 11:40:00

0 COMMENTS

We’d like to hear from you!