Metadata Fields

Every now and then you may find useful to add metadata to parsed data elements. For example, if the source data contains several different record types you may want to include an explicit record type field or the position of a data element in the input stream. To achieve this the pattern language allows to define constant values or matched token positions in patterns.

<pos>:export_name

output type:LONG
quantifier:none
configuration:none

Keyword “pos” (or “POS”) placed between angle brackets defines a meta field capturing the position of a data element matched by the next matcher expression in the input stream. You must assign an export name to expose the metadata field in the resulting query stream.

Note

The position info is available for plaintext and Sxgzip Compression Utilities compressed files. See more in Compressed data

Example: Extracting integers and their position in the input data:

1
2
3
4
5

Pattern:

1
<pos>:position INT:i EOL
position i _unmatched
0 1 NULL
2 2 NULL
4 3 NULL
6 4 NULL
8 5 NULL

<string_value>:export_name

A string (enclosed in single or double quotes) placed between angle brackets defines meta field with constant string_value. This allows assigning record types when parsing transactions consisting of multiple events.

output type:STRING
quantifier:none
configuration:none

Example: Assign record types for request and response events:

2016-03-14 23:37:06 request: type=GET uri="/?a=0&b=apply"
2016-03-14 23:37:07 response: uid=279956825 resp_code=200

Pattern:

1
2
3
4
5
$request = <'req'>:rec_type   TIMESTAMP:timedate ' request:' LD:message EOL;

$response = <'resp'>:rec_type TIMESTAMP:timedate ' response:' LD:message EOL;

($request | $response)

where:

  • line 1 defines subpattern for request record with rec_type meta field having value “req”
  • line 3 defines subpattern for response record with rec_type meta field having value “resp”
  • line 5 is the main pattern statement with alternate group matching either request or response record
rec_type timedate message _unmatched
req 2016-03-14 23:37:06.000 +0000 type=GET uri=’/?a=0&b=apply’ NULL
resp 2016-03-14 23:37:07.000 +0000 uid=279956825 resp_code=200 NULL