The query developer is where users write queries for their data, queries can be saved into the resource tree, downloaded and shared. The query language is based on SQL and is explained here,
The query developer is broken into two sections, the top section is for writing queries and the bottom section is broken down into 3 tabs for viewing query results, query process explanation, and the query log.
CTRL+E`- Execute the query
CTRL+P- Display selected rows of result in selectable format (CSV, TSV, JSON, …). Intended to be used for copy/paste.
CTRL+S- Save query script
CTRL+SPACE- Displays context help on the query and parse language commands and syntax.
Top Section - Query Developer¶
AUTO COMPLETE - Pressing
space bar after any term will show a drop-down menu with definition and
usage information. The same function assists with writing patterns.
In the query developer, users write queries that SpectX uses to parse data. The top menu bar has 5 buttons relevant to query development:
Run- Executes the query (
Chart- Shows various charting and mapping options
Follow- opens new Query tab for querying current result set. This is handy for exploring parsed data set with various different queries.
Save- allows saving current query script or result set as a table within SpectX or export it in different formats.
Completion report bar¶
completion report bar displays vital information about the success of each query. If the query was successful
the data display will be limited. However, broken or corrupted input data cause the
completion report bar to turn
yellow and show what percentage of data was successfully passed. Input data that produced errors are shown in the
query log tab, individual tuples can be double-clicked to display further information.
Query Execution Mode¶
Normally the queries are executed in interactive mode - when the user logs out or client (web browser) connection is dropped for more than a minute during the execution, then query processing gets canceled.
The opposite of that is batch_mode - query is executed regardless of client connection state. This is useful in case of long-running queries storing results.
The execution mode can be set by query configuration parameter query.batch_mode, in query init block. See details here.
Ad hoc vs. Regular Queries¶
The preparation phase when analyzing new data is often long and complicated. You have to get access to data, import it to a system, discover the structure, do transformations and consider necessary enrichments even before starting to think about your analysis questions. SpectX makes this phase smooth and flexible in order to quickly conduct ad-hoc analysis. There are many features to support this: the ability to include data from very different storage locations, no need to worry about importing the data, automated discovery and interactive pattern developer for identifying structure in the underlying data, etc.
However, the queries you make on a regular basis are no less important. What are the features offered by SpectX here?
When making regular queries on data with a known structure, it would be nice not to have to specify the pattern and location of source data for each query. This is what SpectX offers with the views. It feels and looks like writing a query in a relational database. Capturing pattern and location in a view also allow role separation between source data management, data structure definition, and analytics.
SpectX also provides an API for automating execution and integration with other applications.
Last but not least, when you’re dealing with log management in a large organization and experiencing long implementation cycles of changes in the structure of logs, you might want to think about rearranging the log management process. SpectX’s real-time data extract and transformation offer possibilities for eliminating bottlenecks in the data preparation pipeline. Read more in this whitepaper.
Timezones come into play on two occasions: a) when the time info enters into SpectX (i.e when parsing time fields) and when it leaves SpectX (i.e timestamps are being displayed or outputted via the API).
During parsing, timezone information may or may not be present in the time field. The default behavior, in this case, is to use a timezone from the field or a default timezone (UTC). In case you need any other timezone to be used, you need to specify this as an argument to TIMESTAMP matcher. See more at parsing Date and Time.
When outputting to SpectX browser, based UI or API timestamps are converted to strings using timezone according to User Account Properties Timezone setting.
See also the case when time field does not have year here.
Errors encountered during processing queries and patterns are displayed in red message boxes. These must be closed before you can continue with work at hand.