Query Developer

The query developer is where users write queries for their data, queries can be saved into the resource tree, downloaded and shared. The query language is based on SQL and is explained here,

The query developer is broken into two sections, the top section is for writing queries and the bottom section is broken down into 3 tabs for viewing query results, query process explanation, and the query log.

Keyboard shortcuts

  • CTRL+E`- Execute the query
  • CTRL+P - Display selected rows of result in selectable format (CSV, TSV, JSON, …). Intended to be used for copy/paste.
  • CTRL+S - Save query script
  • CTRL+SPACE - Displays context help on the query and parse language commands and syntax.

Top Section - Query Developer

Hint

AUTO COMPLETE - Pressing CTRL + space bar after any term will show a drop-down menu with definition and usage information. The same function assists with writing patterns.

The SpectX query developer top bar with options for running queries, inserting functions, charting and following displayed.

In the query developer, users write queries that SpectX uses to parse data. The top menu bar has 5 buttons relevant to query development:

  • Run - Executes the query (Ctrl + e)
  • Chart - Shows various charting and mapping options
  • Follow - opens new Query tab for querying current result set. This is handy for exploring parsed data set with various different queries.
  • Save - allows saving current query script or result set as a table within SpectX or export it in different formats.
  • Insert - allows generating some commonly used query snippets (such as Timestamps, init block and JavaScript functions).
SpectX Web UI displaying the query developer function.

Bottom section - results viewer

The results pane is where the results of queries are displayed. Depending on the type of query the results pane may have several tabs displaying different information. Click any result in the results tab to access detailed information about that result.

results of networking query

SpectX also displays an explanation tab where you can find a detailed breakdown of the actions performed by SpectX to deliver the results. The explanation tab is useful for troubleshooting as it does not contain any identifiable information from the search data, consequently, it may be shared with SpectX support and 3rd parties.

The query log tab is where SpectX displays files that have caused errors when executing a query. For further information about why the errors occurred, simply double click on an entry in the query log tab and details of the error, including broken or corrupted sections of the source data, will be displayed.

Completion report bar

The completion report bar displays vital information about the success of each query. If the query was successful the data display will be limited. However, broken or corrupted input data cause the completion report bar to turn yellow and show what percentage of data was successfully passed. Input data that produced errors are shown in the query log tab, individual tuples can be double-clicked to display further information.

completion report for corrupted input data

Query Execution Mode

Normally the queries are executed in interactive mode - when the user logs out or client (web browser) connection is dropped for more than a minute during the execution, then query processing gets canceled.

The opposite of that is batch_mode - query is executed regardless of client connection state. This is useful in case of long-running queries storing results.

The execution mode can be set by query configuration parameter query.batch_mode, in query init block. See details here.

Ad hoc vs. Regular Queries

The preparation phase when analyzing new data is often long and complicated. You have to get access to data, import it to a system, discover the structure, do transformations and consider necessary enrichments even before starting to think about your analysis questions. SpectX makes this phase smooth and flexible in order to quickly conduct ad-hoc analysis. There are many features to support this: the ability to include data from very different storage locations, no need to worry about importing the data, automated discovery and interactive pattern developer for identifying structure in the underlying data, etc.

However, the queries you make on a regular basis are no less important. What are the features offered by SpectX here?

When making regular queries on data with a known structure, it would be nice not to have to specify the pattern and location of source data for each query. This is what SpectX offers with the views. It feels and looks like writing a query in a relational database. Capturing pattern and location in a view also allow role separation between source data management, data structure definition, and analytics.

SpectX also provides an API for automating execution and integration with other applications.

Last but not least, when you’re dealing with log management in a large organization and experiencing long implementation cycles of changes in the structure of logs, you might want to think about rearranging the log management process. SpectX’s real-time data extract and transformation offer possibilities for eliminating bottlenecks in the data preparation pipeline. Read more in this whitepaper.

Understanding Timezones

Timezones come into play on two occasions: a) when the time info enters into SpectX (i.e when parsing time fields) and when it leaves SpectX (i.e timestamps are being displayed or outputted via the API).

During parsing, timezone information may or may not be present in the time field. The default behavior, in this case, is to use a timezone from the field or a default timezone (UTC). In case you need any other timezone to be used, you need to specify this as an argument to TIMESTAMP matcher. See more at parsing Date and Time.

When outputting to SpectX browser, based UI or API timestamps are converted to strings using timezone according to User Account Properties Timezone setting.

See also the case when time field does not have year here.

Error Messages

Errors encountered during processing queries and patterns are displayed in red message boxes. These must be closed before you can continue with work at hand.