Pattern Developer

SpectX Web UI displaying the pattern matching function.

The pattern developer is for writing patterns to match and extract data from human-readable files such as .txt HTML or Syslog. Binary formats such as PCAP and ROSBAG are also supported, SpectX has an inbuilt pattern which will automatically be applied to these files.

The pattern developer has 2 sections, the top section is for writing patterns and the bottom section is broken down into 3 tabs for editing source data, viewing parse results and previewing how the pattern will treat the source data.

A detailed guide on SpectX pattern language including many step-by-step guides can be found here, .. there is also a cheat sheet with frequently used expressions and their meanings.

To write a pattern for a specific data set:

  1. Click data browser.
  2. Navigate to the data.
  3. Click prepare pattern.
  4. Write/alter the pattern in the top section of the pattern developer.

Top Section - Pattern Writing

Hint

AUTO COMPLETE - Pressing CTRL + space bar after any term will show a drop-down menu with definition and usage information. The same function assists with writing patterns.

As the pattern is written the matched elements of the source data are highlighted in the Parse Preview tab in the bottom section

Bottom Section - Source Data Editor, Parse Results Viewer, Parse Preview

The bottom half of the pattern developer has 3 tabs:

  • Data Editor - Displays a snippet of the source data selected form the Input Data Browser.
  • Results - Shows how the raw file data will be presented if the pattern in the top section is applied to it.
  • Parse Preview - Visually represents how the pattern is matched to raw data using shading, yellow for first, blue for second. As the pattern is written the data divides into contrasting bars. If the pattern fails to capture all of the raw data gaps will be visible.

When to use the pattern developer

Hint

You do not have to match every field in a set of data to use SpectX, for example, if you are only interested in extracting the ISP address, username, country code, and login time from an access log, you can write a pattern extracting only the fields relevant to you.

It is not necessary to use the pattern developer each time data is queried, SpectX can query partially matched data, when performing ad hoc analysis it is often more convenient to match and extract only the fields relevant to the query being performed.

When attempting to match data to patterns SpectX first scans the resource tree for all available pattern files (.sxp) then uses fuzzy logic to match the input data to a pattern. Ultimately SpectX selects the best fit based on the percentage match. For this reason, only complete patterns should be saved in the resource tree.

When and where to save patterns

Hint

Do not save every pattern that you write in the resource tree. Poor quality, incomplete, or overly general patterns will cause the automatic pattern recognition to produce inaccurate or erratic matches.

When regularly querying data that SpectX can not automatically recognize it is useful to write a custom pattern and save it in the resource tree, after which, SpectX will automatically match the pattern to the data.

SpectX searches folders in the following order:

  1. User
  2. Shared
  3. System

Securing patterns

In general, patterns do not represent a significant security issue, however, there are two security concerns that should be considered by system admins.

Accidental modification - users may accidentally alter or delete patterns in the shared drive. To mitigate this risk, the important patterns should be stored in the system folder and backed up regularly.

Reverse engineering - a user may be able to determine what level and type of logging are used in certain areas of a system by observing the fields that a pattern includes. This issue is addressed by keeping sensitive patterns and queries in the system folder and redacting the field from the query output. detailed guides on securing data, queries, and patterns can be found in the securing section.

Sharing patterns

Patterns can be shared between different SpectX installations by downloading them to local storage and manually uploading them to the resource tree. Patterns can also be copied and sent in plain text format. SpectX has inbuilt protections that prevent malicious patterns and scripts from being executed in it.