Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Edge Data Streaming
  3. Licenses
  4. Using Informatica Administrator
  5. Creating and Managing the Edge Data Streaming Service
  6. Edge Data Streaming Entity Types
  7. Edge Data Streaming Nodes
  8. Data Connections
  9. Working With Data Flows
  10. Managing the Edge Data Streaming Components
  11. Security
  12. High Availability
  13. Disaster Recovery
  14. Monitoring Edge Data Streaming Entities
  15. Appendix A: Troubleshooting
  16. Appendix B: Frequently Asked Questions
  17. Appendix C: Regular Expressions
  18. Appendix D: Command Line Program
  19. Appendix E: Configuring Edge Data Streaming to Work With a ZooKeeper Observer
  20. Appendix F: Glossary

User Guide

User Guide

Pattern Examples

Pattern Examples

The following examples describe some patterns that you can configure:

Pattern Without Custom Regular Expression

If you want to transform the input
vds
into JSON format, configure the following properties:
  • Entity Name. Enter a name for the transformation.
  • Pattern. Enter
    %{WORD:myword}
When you deploy the data flow, you get the following output:
{"myword": [["vds"]]}

Pattern With Custom Regular Expression

If you want to transform the input
ABC
into JSON format, configure the following properties:
  • Entity Name. Enter a name for the transformation.
  • Pattern. Enter
    %{ALLCAPS:capword}
  • Custom Regex. Enter
    ALLCAPS=[A-Z]+
When you deploy the data flow, you get the following output:
{"capword":"ABC"}

Pattern to Convert a Log Line

If you want to transform the log line
Jul 9 22:41:51 myserver sshd[4295]: Failed password for invalid user myuser from 220.113.135.154 port 55993 ssh2
into JSON format, configure the following properties:
  • Entity Name. Enter a name for the transformation.
  • Pattern. Enter
    %{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:host_target} sshd\[%{BASE10NUM}\]: Failed password for (invalid user |)%{USERNAME:username} from %{IP:src_ip} port %{BASE10NUM:port} ssh2
When you deploy the data flow, you get the following output:
{"BASE10NUM":4295,"HOUR":22,"MINUTE":41,"MONTH":"Jul","MONTHDAY":9,"SECOND":51,"TIME":"22:41:51","host_target":"myserver","port":55993,"src_ip":"220.113.135.154","timestamp":"Jul 9 22:41:51","username":"myuser"}

Pattern to Convert a Syslog Log Line

If you want to transform the log line
Oct 17 08:59:00 suod newsyslog[6215]: logfile turned over
into JSON format, configure the following properties:
  • Entity Name. Enter a name for the transformation.
  • Pattern. Enter
    %{SYSLOGBASE} %{GREEDYDATA}
When you deploy the data flow, you get the following output:
{"SYSLOGBASE":[["Oct 17 08:59:00 suod newsyslog[6215]:"]],"timestamp":[["Oct 17 08:59:00"]],"MONTH":[["Oct"]],"MONTHDAY":[["17"]],"TIME":[["08:59:00"]],"HOUR":[["08"]],"MINUTE":[["59"]],"SECOND":[["00"]],"SYSLOGFACILITY":[[null]],"facility":[[null]],"priority":[[null]],"logsource":[["suod"]],"IPORHOST":[["suod"]],"HOSTNAME":[["suod"]],"IP":[[null]],"IPV6":[[null]],"IPV4":[[null]],"SYSLOGPROG":[["newsyslog[6215]"]],"program":[["newsyslog"]],"pid":[["6215"]],"GREEDYDATA":[["logfile turned over"]]}

Pattern to Convert an Apache Log Line

If you want to transform the log line
127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326
into JSON format, configure the following properties:
  • Entity Name. Enter a name for the transformation.
  • Pattern. Enter
    %{COMMONAPACHELOG}
When you deploy the data flow, you get the following output:
{"COMMONAPACHELOG":[["127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET/apache_pb.gifHTTP/1.0" 200 2326"]],"clientip":[["127.0.0.1"]],"HOSTNAME":[["127.0.0.1"]],"IP":[[null]],"IPV6":[[null]],"IPV4":[[null]],"ident":[["-"]],"USERNAME":[["-","frank"]],"auth":[["frank"]],"timestamp":[["10/Oct/2000:13:55:36 -0700"]],"MONTHDAY":[["10"]],"MONTH":[["Oct"]],"YEAR":[["2000"]],"TIME":[["13:55:36"]],"HOUR":[["13"]],"MINUTE":[["55"]],"SECOND":[["36"]],"INT":[["-0700"]],"verb":[["GET"]],"request":[["/apache_pb.gif"]],"httpversion":[["1.0"]],"BASE10NUM":[["1.0","200","2326"]],"rawrequest":[[null]],"response":[["200"]],"bytes":[["2326"]]}

Pattern to Convert a log4j Log Line

If you want to transform the log line
2015-03-03 11:35:53,759 [WARN ] [lbm:Thread-27] Core-5688-1883: timer returned error 5 [CoreApi-5688-3337: lbm_socket_sendb send/sendto: (10049) The requested address is not valid in its context.
into JSON format, configure the following properties:
  • Entity Name. Enter a name for the transformation.
  • Pattern. Enter
    %{TIMESTAMP_ISO8601} \[%{LOGLEVEL} %{GREEDYDATA}
When you deploy the data flow, you get the following output:
{"TIMESTAMP_ISO8601":[["2015-03-03 11:35:53,759"]],"YEAR":[["2015"]],"MONTHNUM":[["03"]],"MONTHDAY":[["03"]],"HOUR":[["11",null]],"MINUTE":[["35",null]],"SECOND":[["53,759"]],"ISO8601_TIMEZONE":[[null]],"LOGLEVEL":[["WARN"]],"GREEDYDATA":[["] [lbm:Thread-27] Core-5688-1883: timer returned error 5 [CoreApi-5688-3337: lbm_socket_sendb send/sendto: (10049) The requested address is not valid in its context."]]}

0 COMMENTS

We’d like to hear from you!