Table of Contents

  1. Preface
  2. Introduction to Informatica Edge Data Streaming
  3. Licenses
  4. Using Informatica Administrator
  5. Creating and Managing the Edge Data Streaming Service
  6. Edge Data Streaming Entity Types
  7. Edge Data Streaming Nodes
  8. Data Connections
  9. Working With Data Flows
  10. Managing the Edge Data Streaming Components
  11. Security
  12. High Availability
  13. Disaster Recovery
  14. Monitoring Edge Data Streaming Entities
  15. Troubleshooting
  16. Frequently Asked Questions
  17. Regular Expressions
  18. Command Line Program
  19. Configuring Edge Data Streaming to Work With a ZooKeeper Observer
  20. Glossary

User Guide

User Guide

Pattern Examples

The following examples describe some patterns that you can configure:
If you want to transform the input vds into JSON format, configure the following properties:
  • Entity Name. Enter a name for the transformation.
  • Pattern. Enter %{WORD:myword}
When you deploy the data flow, you get the following output:
{"myword": [["vds"]]}
If you want to transform the input ABC into JSON format, configure the following properties:
  • Entity Name. Enter a name for the transformation.
  • Pattern. Enter %{ALLCAPS:capword}
  • Custom Regex. Enter ALLCAPS=[A-Z]+
When you deploy the data flow, you get the following output:
{"capword":"ABC"}
If you want to transform the log line Jul 9 22:41:51 myserver sshd[4295]: Failed password for invalid user myuser from 220.113.135.154 port 55993 ssh2 into JSON format, configure the following properties:
  • Entity Name. Enter a name for the transformation.
  • Pattern. Enter %{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:host_target} sshd\[%{BASE10NUM}\]: Failed password for (invalid user |)%{USERNAME:username} from %{IP:src_ip} port %{BASE10NUM:port} ssh2
When you deploy the data flow, you get the following output:
{"BASE10NUM":4295,"HOUR":22,"MINUTE":41,"MONTH":"Jul","MONTHDAY":9,"SECOND":51,"TIME":"22:41:51","host_target":"myserver","port":55993,"src_ip":"220.113.135.154","timestamp":"Jul 9 22:41:51","username":"myuser"}
If you want to transform the log line Oct 17 08:59:00 suod newsyslog[6215]: logfile turned over into JSON format, configure the following properties:
  • Entity Name. Enter a name for the transformation.
  • Pattern. Enter %{SYSLOGBASE} %{GREEDYDATA}
When you deploy the data flow, you get the following output:
{"SYSLOGBASE":[["Oct 17 08:59:00 suod newsyslog[6215]:"]],"timestamp":[["Oct 17 08:59:00"]],"MONTH":[["Oct"]],"MONTHDAY":[["17"]],"TIME":[["08:59:00"]],"HOUR":[["08"]],"MINUTE":[["59"]],"SECOND":[["00"]],"SYSLOGFACILITY":[[null]],"facility":[[null]],"priority":[[null]],"logsource":[["suod"]],"IPORHOST":[["suod"]],"HOSTNAME":[["suod"]],"IP":[[null]],"IPV6":[[null]],"IPV4":[[null]],"SYSLOGPROG":[["newsyslog[6215]"]],"program":[["newsyslog"]],"pid":[["6215"]],"GREEDYDATA":[["logfile turned over"]]}
If you want to transform the log line 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 into JSON format, configure the following properties:
  • Entity Name. Enter a name for the transformation.
  • Pattern. Enter %{COMMONAPACHELOG}
When you deploy the data flow, you get the following output:
{"COMMONAPACHELOG":[["127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET/apache_pb.gifHTTP/1.0" 200 2326"]],"clientip":[["127.0.0.1"]],"HOSTNAME":[["127.0.0.1"]],"IP":[[null]],"IPV6":[[null]],"IPV4":[[null]],"ident":[["-"]],"USERNAME":[["-","frank"]],"auth":[["frank"]],"timestamp":[["10/Oct/2000:13:55:36 -0700"]],"MONTHDAY":[["10"]],"MONTH":[["Oct"]],"YEAR":[["2000"]],"TIME":[["13:55:36"]],"HOUR":[["13"]],"MINUTE":[["55"]],"SECOND":[["36"]],"INT":[["-0700"]],"verb":[["GET"]],"request":[["/apache_pb.gif"]],"httpversion":[["1.0"]],"BASE10NUM":[["1.0","200","2326"]],"rawrequest":[[null]],"response":[["200"]],"bytes":[["2326"]]}
If you want to transform the log line 2015-03-03 11:35:53,759 [WARN ] [lbm:Thread-27] Core-5688-1883: timer returned error 5 [CoreApi-5688-3337: lbm_socket_sendb send/sendto: (10049) The requested address is not valid in its context. into JSON format, configure the following properties:
  • Entity Name. Enter a name for the transformation.
  • Pattern. Enter %{TIMESTAMP_ISO8601} \[%{LOGLEVEL} %{GREEDYDATA}
When you deploy the data flow, you get the following output:
{"TIMESTAMP_ISO8601":[["2015-03-03 11:35:53,759"]],"YEAR":[["2015"]],"MONTHNUM":[["03"]],"MONTHDAY":[["03"]],"HOUR":[["11",null]],"MINUTE":[["35",null]],"SECOND":[["53,759"]],"ISO8601_TIMEZONE":[[null]],"LOGLEVEL":[["WARN"]],"GREEDYDATA":[["] [lbm:Thread-27] Core-5688-1883: timer returned error 5 [CoreApi-5688-3337: lbm_socket_sendb send/sendto: (10049) The requested address is not valid in its context."]]}


Updated March 19, 2019


Explore Informatica Network