JSONified Windows Event Log

Windows Event log collected to central storage using the Fluentd windows_eventlog plugin is an example of a record structure with a collection metadata header and JSON formatted event data:

Example:

2019-01-29T11:20:04+00:00  DESKTOP-VHONLD1.winevt {"channel":"application",
                                                     "record_number":"7",
                                                     "time_generated":"2019-01-24 17:26:05 -0800",
                                                     "time_written":"2019-01-24 17:26:05 -0800",
                                                     "event_id":"1034",
                                                     "event_type":"information",
                                                     "event_category":"0",
                                                     "source_name":"Software Protection Platform Service",
                                                     "computer_name":"WIN-3C2LFLGB5E2",
                                                     "user":"",
                                                     "description":"Duplicate definition of policy found. Priority=100\r\n",
                                                     "string_inserts":["AAD-WindowsCore-AddAccountRestrictions","100"]}

Hint

You can find sample log file by navigating with Input Data Browser to s3s://spectx-docs/formats/log/winevt-json/2019-01-29.DESKTOP-VHONLD1.winevt.log.sx.gz

Parse

Parsing such records is fairly simple. We could just use the TIMESTAMP, LD and JSON matchers, separated by tab character.

However, as Windows event contains a fixed number of common fields and variable data is contained in the string_inserts array, we can export all these fields to resultset columns converting them to respective types:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
TIMESTAMP('yyyy-MM-ddTHH:mm:ssZ'):time_collected '\t'
LD:host '\t'
JSON{
 STRING:channel,
 INT:record_number,
 TIMESTAMP:time_generated,
 TIMESTAMP:time_written,
 INT:event_id,
 INT:event_category,
 STRING:source_name,
 STRING:computer_name,
 STRING:user,
 STRING:description,
 JSON_ARRAY{}(typed=true):string_inserts
}(flat=true)
EOL

where:

  • lines 1 and 2 extracts the collection metadata header fields
  • lines 4 to 13 extract common fields of event
  • line 14 extracts variable fields as ARRAY
  • line 15 instructs JSON matcher to export all its parsed fields directly to resultset columns
  • line 16 matches line break (LF) which terminates our record
timestamp host channel record_number time_generated time_written event_id event_category source_name computer_name user description string_inserts
2019-01-29 11:20:04.000 +0000 DESKTOP-VHONLD1.winevt application 7 2019-01-24 17:26:05.000 +0000 2019-01-24 17:26:05.000 +0000 1034 0 Software Protection Platform Service WIN-3C2LFLGB5E2   Duplicate definition of policy found. Policy name=AAD-WindowsCore-AddAccountRestrictions Priority=100 [AAD-WindowsCore-AddAccountRestrictions, 100]

Query

1
2
3
4
LIST(src:'s3s://spectx-docs/formats/log/winevt-json/2019-01-29.DESKTOP-VHONLD1.winevt.log.sx.gz')
| PARSE(pattern:FETCH('https://raw.githubusercontent.com/spectx/resources/master/examples/patterns/winevt-json/winevt-json.sxp'))
| filter (event_id = 1 AND event_category = 5)
;

Hint

You can download full code of the pattern and query at https://github.com/spectx/resources/tree/master/examples/patterns/winevt-json