Java Debug Log

A Java debug log consists of one type of events (debug messages) which can span across multiple lines when the message contains an exception stack trace.

Example:

2015-10-03 16:32:50.009 +0000    INFO    connecting to db ...
2015-10-03 16:32:51.034 +0000    ERROR   com.spectx.webconsole.jsp.data.SQLTimeSeriesCache -- SQLTimeSeries remote fetch failed
org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
        at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
        at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
        at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
        at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)
        at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
        at org.postgresql.Driver.makeConnection(Driver.java:393)
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
        at java.net.Socket.connect(Socket.java:529)
        at java.net.Socket.connect(Socket.java:478)
        at java.net.Socket.<init>(Socket.java:375)
        at java.net.Socket.<init>(Socket.java:189)
        at org.postgresql.core.PGStream.<init>(PGStream.java:62)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:76)
        ... 23 more
2015-10-03 16:32:54.844 +0000    DEBUG   closing socket
2015-10-03 16:32:56.232 +0000    INFO    connecting to db ...

Hint

You can find the sample of SpectX debug log by navigating with Input Data Browser to s3s://spectx-docs/formats/log/log4j/spectx-1.40-debug.log

Parse

The pattern to parse this:

1
2
3
4
5
6
7
8
9
$hdr =                                  //record header:
TIMESTAMP('yyyy-MM-dd HH:mm:ss.SSS Z'):time   //timestamp
SPACE                                   //followed by by one or more spaces
UPPER:level                             //followed by uppercase log level
;

$hdr                                    //each record begins with header
DATA{1,50000}:message                   //and is followed by message with one or more lines up to 50Kb
(EOL (>>$hdr | EOF))                    //until we see the EOL followed by header or end of file

See also

the detailed walk-through of the pattern at Parsing Multiline Records

Query

Let’s find all records where the message contains exception stack trace from the sample SpectX debug log:

1
2
3
4
LIST(src:'s3s://spectx-docs/formats/log/log4j/spectx-1.40-debug.log')
| parse(pattern:FETCH('https://raw.githubusercontent.com/spectx/resources/master/examples/patterns/log4j/log4j.sxp'))
| filter(message contains 'Exception')
;

where line:

  1. Performs listing source data file
  2. Retrieves the content source data file and parses it according to the pattern specified.
  3. Retains only those records containing the word “Exception” in the resultset.
timestamp level thread class message
2019-12-04 08:19:48.782 +0000 DEBUG qtp100929741-805 c.s.w.a.a.pattern.PatternActionView User error LD must b …

Hint

You can download full code of the pattern at https://github.com/spectx/resources/tree/master/examples/patterns/log4j