filter

The filter command filters the resultset using expression returning a boolean value (boolean_expression). It may consist of multiple boolean expressions combined using AND, OR NOT operators.

Most common filtering / searching examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
| filter(_raw_text like '%/vpns/%')
| filter(type = 'logout')
| filter(last_modified >= now()[-7 day])
| filter(queryTime BETWEEN T('2020-01-21 12:22:42.061 +0200') AND now())
| filter(email in ('user1@m0mbo.org', 'admin@sumb0.net'))
| filter(href not like '%/v2/spectx-core/%')
| filter(ip not in 192.168.0.0/16)
| filter(dst = 8.8.8.8 AND src = 192.168.2.2)
| filter(cc(ip) = 'EE')
| filter(string({*}) like '%/vpns/%')
| filter(agent not like 'CitrixRec%')
| filter(lower(headers[User_Agent]) not like '%mozilla%')
| filter(<boolean_expression>)

Description

  1. Keeps records where _raw_text (source raw record text) contains string “/vpns/”
  2. Keeps records where string field type is “logout”
  3. Keeps records where timestamp field last_modified is greater then 7 days ago.
  4. Keeps records where timestamp field queryTime is between defined time and current time.
  5. Keeps records where string field email is one of emails from list.
  6. Discards records where string field href contains “/v2/spectx-core/”
  7. Discards records where ip is from internal network (192.168.0.0/16)
  8. Keeps records where dst is google dns server AND src is 192.168.2.2.
  9. Keeps records where ip is from Estonia using GeoIp lookup.
  10. Keeps records where ANY field contains string “/vpns/”
  11. Discards records where string field agent begins with “CitrixRec”
  12. Discards records where inner field User_Agent contains “mozilla” (case insensitive)
  13. Keeps records where your expression matches.

Example:

1
dual(10) | filter (i > 5 AND i < 8);
1
2
3
4
5
6
7
$simple_apache_format="
  IPADDR:ip DATA TIMESTAMP('[dd/MMM/yyyy:HH:mm:ss Z] '):time
  DQS:query ' ' INT:status ' ' LONG:bytes ' ' DQS:ref ' ' DQS:browser EOL
";

LIST('s3s://spectx-docs/formats/log/apache/apache_access.log.sx.gz')
| parse($simple_apache_format)
1
2
3
4
5
6
7
8
$simple_apache_format="
  IPADDR:ip DATA TIMESTAMP('[dd/MMM/yyyy:HH:mm:ss Z] '):time
  DQS:query ' ' INT:status ' ' LONG:bytes ' ' DQS:ref ' ' DQS:browser EOL
";

LIST('s3s://spectx-docs/formats/log/apache/apache_access.log.sx.gz')
| parse($simple_apache_format)
| filter(_raw_text like '%phpmyadmin%')