The pattern developer is for writing patterns to match and extract data from human-readable files such as .txt HTML or Syslog. Binary formats such as PCAP and ROSBAG are also supported, SpectX has an inbuilt pattern which will automatically be applied to these files.
The pattern developer has 2 sections, the top section is for writing patterns and the bottom section is broken down into 3 tabs for editing source data, viewing parse results and previewing how the pattern will treat the source data.
A detailed guide on SpectX pattern language including many step-by-step guides can be found here.
To write a pattern for a specific data set:
- Navigate to the data.
- Write/alter the pattern in the top section of the pattern developer.
Top Section - Pattern Writing¶
AUTO COMPLETE - Pressing
space bar after any term will show a drop-down menu with definition and
usage information. The same function assists with writing patterns.
As the pattern is written the matched elements of the source data are highlighted in the
Parse Preview tab in the
When to use the pattern developer¶
You do not have to match every field in a set of data to use SpectX, for example, if you are only interested in extracting the ISP address, username, country code, and login time from an access log, you can write a pattern extracting only the fields relevant to you.
It is not necessary to use the pattern developer each time data is queried, SpectX can query partially matched data, when performing ad hoc analysis it is often more convenient to match and extract only the fields relevant to the query being performed.
When attempting to match data to patterns SpectX first scans the resource tree for all available pattern files (.sxp) then uses fuzzy logic to match the input data to a pattern. Ultimately SpectX selects the best fit based on the percentage match. For this reason, only complete patterns should be saved in the resource tree.
When and where to save patterns¶
Do not save every pattern that you write in the resource tree. Poor quality, incomplete, or overly general patterns will cause the automatic pattern recognition to produce inaccurate or erratic matches.
When regularly querying data that SpectX can not automatically recognize it is useful to write a custom pattern and save it in the resource tree, after which, SpectX will automatically match the pattern to the data.
SpectX searches folders in the following order:
In general, patterns do not represent a significant security issue, however, there are two security concerns that should be considered by system admins.
Accidental modification - users may accidentally alter or delete patterns in the shared drive. To mitigate this risk, the important patterns should be stored in the system folder and backed up regularly.
Reverse engineering - a user may be able to determine what level and type of logging are used in certain areas of a system by observing the fields that a pattern includes. This issue is addressed by keeping sensitive patterns and queries in the system folder and redacting the field from the query output. detailed guides on securing data, queries, and patterns can be found in the securing section.