Release Notes

v1.4.77 (2021-06-16)

  • Root directory chooser dialog in file:// datastore configuration window on Windows does not show directory tree.
  • Handling of paths containing special characters for JDBC SQLite connectivity.

v1.4.76 (2021-05-14)

  • Failure to read previously saved index data by raw cursor does not abandon the query any more. The cursor gets reset and query continues, and a warning message is printed to both error and query log.
  • Support for ZStandard decompression in PARSE (.zst files produced with zstd utility)
  • Handling of UNC names on Windows.
  • export() format text/sx-raw-bytes fails.

v1.4.75 (2021-04-21)

  • Record details view - render VAR<X> detail view the same way as X.
  • Record details - add filtering operations to context menu.
  • Operating system commands execution via exec:// protocol.
  • EVTX command for processing Windows XML Event Log (EVTX) files.
  • Editor autocomplete - unhandled code assist hotkeys ruin code completion.

v1.4.74 (2021-04-05)

  • Support for BusyBox userland ssh:// datastores with disabled SFTP access.
  • Optional userland selection for ssh:// datastores with disabled SFTP access.
  • SpectXDesktop: example patterns are not created on the first run.

v1.4.73 (2021-03-31)

  • Elasticsearch - respect http.nonProxyHosts when processing http(s).proxyHost & http(s).proxyPort system properties.
  • Elasticsearch ES_QUERY:
    • support nested fields for first argument
    • better error messages for mismatching argument types (first argument should be a field)
  • Editor autocomplete - improved completion of invocation parameters.
  • msapi:// and gsuite:// protocols availability in SpectX with Free license is extended until 1st July 2021.
  • Config key query.parse.preLock to PARSE for immediate locking of listed files by SourceAgent. This feature requires SourceAgent version 1.4.73 or above.
  • Possibilities to access HTTP response headers and to manipulate with HTTP request method, headers and body by changing configuration in queries and data store definition files.
  • Elasticsearch - configurable timeouts for ES and SAVE_ES via,, and query.save_es.connect_timeout, query.save_es.socket_timeout, query.save_es.max_retry_timeout options.
  • Boolean switches ignoreNotFoundErrors and ignoreUnknownHostErrors to FETCH function.
  • Options “Use Sudo” and “Disable SFTP” to ssh:// datastore configuration.
  • Impossible to log in to SpectX using Google OAuth in Safari and Firefox.
  • SpectX Server, installed as a Windows service to run under domain account, fails to start due to logon failure.
  • SpectX Server, installed as a Windows service to run under managed service account, fails to start due to missing “Log on as a service” privilege.
  • PARSE may report interrupted IO failure after query has been cancelled.
  • Elasticsearch - filters that use parent[child] field dereferencing are not pushed down to ES when they appear after a select() pipe.
  • Elasticsearch - date and date_nanos fields don’t support floating point values and floating point values encapsulated in strings.
  • Elasticsearch - date and date_nanos fields don’t support variable number of fractional seconds.

v1.4.72 (2021-03-02)

  • CACHE_V1 storage format bug.

v1.4.71 (2021-03-01)

  • Docker - glibc upgraded to 2.33-r0.
  • Allow using mime types for export format parameter.
  • Record details dialog - added option to create a select() statement from selected items to context menu.
  • Resource properties dialog shows producing script path for sxt tables.
  • Elasticsearch - search queries now use source filtering to constrain the set of _source fields returned from ES.
  • Elasticsearch - master-detail mode for querying sparse indexes.
  • Elasticsearch - Index setting max_docvalue_fields_search respected for search queries.

v1.4.70 (2021-02-18)

  • In addition to Personal SpectX license, SpectXDesktop now supports Enterprise and Business SpectX licenses.
  • Presentation options parameters to UDF function TO_JSON.
  • Editor autocomplete - operators.
  • Formats text/sx-separated-text & text/sx-text to API, sx-text & sx-separated-text to export output command.
  • Display rows as dialog, format - allow search by typing, remember choice.
  • Record details dialog - added context menu with expand, expand deep, collapse and select children actions; operates on selected items.
  • Coordinate ordering in JDBC PostgreSQL POINT & GEOMETRY reading and writing was swapped. PostgreSQL uses (lon, lat), SpectX uses (lat, lon). Also emphasized the coordinate order in docs.
  • Elasticsearch - typeless index misidentified as legacy index under certain conditions.
  • Elasticsearch - filter pushdown for operators with mismatching operand types.

v1.4.69 (2021-02-01)

  • Symbolic links are now always followed by SpectX when accessing files and directories on remote file systems over ssh protocol, unless configured otherwise for the ssh datastore. Before the change, symbolic links were not followed.
  • Access permission descriptions in ACL box of datastore configuration window now include access type (read, write) specification as last element.
  • msapi:// and gsuite:// protocols availability in SpectX with Free license is extended until 1st April 2021.
  • engine.js.exposed_classes can use * as a wildcard.
  • SpectX now unpacks sqlite and jetty native libraries to directory named “temp” in it’s installation directory on Linux and OSX as, opposed to system-provided temp directories it used before. To change the target directory, use “” system property in SpectX environment script.
  • Root directory chooser dialog to file:// datastore configuration window.
  • Pipe input to GREP input command.
  • export output command.
  • Value of wgui.instanceName configuration parameter to SpectX page title in the browser.
  • Parameter reset to raw cursor.
  • LIKE supports escaping special characters %, _, \ with backslash \.
  • Chart window - Added button to apply selection filter to script.
  • System Status > Running Queries - double clicking on a query shows query script text in a popup.
  • Editor autocomplete - JDBC_* input and process command parameters and configuration settings.
  • JDBC PostgreSQL:
    • support reading POINT type as GEOPOINT into SpectX.
    • support reading PostGIS GEOMETRY type as VARIANT into SpectX - POINT (x y) is read as VAR<GEOPOINT>, other geometries as VAR<STRING>.
    • support writing GEOPOINT as GEOMETRY when PostGIS is available, otherwise as POINT.
  • New UDF function VARIANT_OBJECT_AGG.
  • New UDF function TO_JSON.
  • Option to use spaces instead of tabs for indentation, set the tab size and show whitespace characters in editor.
  • JDBC - Optimized performance by adding parallel mode to JDBC_INSERT, JDBC_UPDATE, JDBC_DELETE, JDBC_MODIFY.
  • JDBC - Optimized PostgreSQL INSERT performance by adding support for COPY .. BINARY mechanism.
  • msapi:// and gsuite:// protocols: with “Hour in File Name” set and “Hour in File Path” unset, hourly files for the first hour of each day get only produced.
  • Charset conversion and/or serial decompression is not applied to multiple small-sized blobs found with globbed uri having content handling instructions in it.
  • Automatically created query for parsing SQLite files escapes too many characters in file path.
  • Editor autocomplete - resource paths matching is case sensitive.
  • Tuple browser - TIMESTAMP & TIMESTAMP_NANO values are not displayed in user timezone.
  • Resultset browser - TIMESTAMP & TIMESTAMP_NANO values are not displayed in user timezone when inside compound types.
  • API - Range headers do not work with GET /scripts/ & POST / requests when parallel serialization is enabled.
  • API - Column order in serialized resultset can differ from script output column order.
  • API - POST / & POST /queries fail to read script from post body when authentication token is passed in query string and content-type is application/x-www-form-urlencoded or multipart/form-data.

v1.4.68 (2020-11-27)

  • “All platforms (no JRE)” package of SpectX Server Edition does not contain “tools” directory with SourceAgent and SxGzip Compression Utility anymore, as these both are available as separate downloads, in both licensee cabinet and SpectX products download page.
  • msapi:// and gsuite:// protocols availability in SpectX with Free license is extended until 1st February 2021.
  • JRE upgraded to 1.8.0_275 in packages for all platforms.
  • Config key wgui.instanceName for displaying configured value under SpectX logo on login page and in main view. Available only in Server Edition.
  • Configuration properties can be defined as environment variables instead of a configuration file or to override certain properties in the configuration file.
  • Editor autocomplete - LIST and PARSE input and process command parameters and configuration settings.
  • Editor autocomplete - stream & expression blocks support:
    • when inside block declaration, complete variables declared in enclosing scopes & parameters of enclosing blocks.
    • for block invocations, complete parameter names.
  • Editor autocomplete - show descriptions for pipe task fields.
  • CREATE FUNCTION JavaScript directive can return VARIANT_OBJECT type.
  • Configurable list of Java classes exposed to CREATE FUNCTION JavaScript directive via engine.js.exposed_classes.
  • Color of a resultset field named xyz can be specified via column _color_xyz value.
  • In Configuration window of the SpectX Desktop, cache directory path obtained by means of a file chooser dialog is not saved in the configuration.
  • On linux, SpectX update via UI fails if SpectX is launched with systemd.
  • Elasticsearch compatibility for versions 8x, <2.0.0, <5.0.0, 5.1.2 - 5.3.2.
  • Editor autocomplete - ES and PARSE commands parameters do not complete when command name is expressed in mixed case.
  • Resultset browser error reporting improved when data files have been manually removed.
  • LIST process command - parameters specified in source stream do not take effect when also selected in output.
  • SpectXDesktop sometimes crashes on OSX when its window is moved from main screen to external monitor screen.

v1.4.67 (2020-11-03)

  • “Change License” sub-menu moved from “Maintenance” menu to “About”
  • LIST() input command - accepts parameters when src is a STRING.
  • Resultset Code Snippet menu - time distribution for TIMESTAMP_NANO columns.
  • Docker image features to support no-interaction deployment:
    • -E/–eula argument shows EULA.
    • SPECTX_ACCEPT_EULA=true environment variable accepts EULA.
    • SPECTX_ADMIN_PASSWORD environment variable specifies password for admin user that doesn’t have to be changed on first login.
  • New UDF function ARRAY_REVERSE.
  • msapi:// and gsuite:// protocols produce an excessive virtual file for overlap hour occurring in transition from DST when datastore is configured to have hour pointer in both file name and path.
  • Configuration key wgui.dataBrowser.showBlobsDisallowedInACL is ignored by data browser.
  • SpectX may crash on Linux if launched with system (not bundled) JRE when processing gzip or bzip2 compressed files.
  • LIST() - tz parameter in src tuple ignored.
  • DUAL() result tuple[ip6] and tuple[ip6s] contain ipv4 addresses.
  • Some keywords were not highlighted in script editor.
  • Data store with execute but no read permission cannot be opened from resource tree.

v1.4.66 (2020-10-15)

  • LIKE supports matching a single character with “_”.
  • Resource api access view without configured token
  • Renaming resource while changing only filename case deletes it
  • Processing of some concatenated gzip files fails on Intel-based 64 bit OSX and Linux platforms.

v1.4.65 (2020-10-05)

  • SAVE_ES() - support writing column types:
    • ARRAY, VARIANT_ARRAY -> multivalued field
    • DURATION-> Long
    • IPNET -> Ip_range
    • TIMESTAMP_NANO -> Date
    • IPSOCKET -> Text
    • MACADDR -> Text
  • ES() - support reading index types:
    • Ip_range -> IPNET
    • Date_nanos -> TIMESTAMP_NANO
  • msapi:// protocol for accessing Azure Active Directory, Office 365 and Azure activity audit logs through Azure Active Directory Activity reports, Office 365 Management Activity and Azure Monitor REST APIs
  • Editor autocomplete - positional call parameters for $() @() @@()
  • Editor autocomplete - show default values for resource stream init parameters
  • SAVE_ES() - Elasticsearch connections are not closed
  • ES() - retrieving string fields from _source when represented as non-strings
  • Resultset browser - HTML injection in column headers
  • API - parallel serialization misbehaves with sort()

v1.4.64 (2020-09-23)

  • Editor autocomplete - suggest resource stream call parameters
  • Editor autocomplete - suggested resources have tooltip with path link
  • New pipe command select_replace
  • New pipe command select_rename
  • New pipe command select_remove
  • Allows extra comma at the end of list. For example | select(field, field2, field3,)
  • WindowsDefender exclusion paths querying fails starting with Windows 10 version 2004

v1.4.63 (2020-09-17)

  • Negative indexes support for SUBSTR function.
  • Export result as CCSV produces a file with .csv extension
  • Support for offset and length arguments in Data Access URI Syntax’s fragment part and compress_type field in the LIST have been removed. Instead, newly introduced slicing and decompression instructions can be used.
  • Resource tree expands too many folders on navigation
  • API - free result table when cancelling query
  • File rename dialog layout garbled for long filenames
  • ‘Display selection as’ with no rows selected displays 0 rows
  • Resource API access tab - limit response preview output size to avoid freezing UI

v1.4.61 (2020-08-24)

  • Copy/Paste functions for files and folders to right-click item menus in Resource Tree.
  • Listing of sx:/ datastore with URIs having glob patterns in directory names of the path returns incomplete results (regression bug since v1.4.57)

v1.4.60 (2020-08-19)

  • Map - do not constrain zoom when features are close together
  • Resource API access tab - response preview
  • SourceAgent sa/sas protocols listing optimisation regression (since v1.4.55)

v1.4.59 (2020-08-14)

  • Markdown table export format
  • Ndjson export format
  • New keyboard shortcut CTRL + / (Toggle line comment)
  • New keyboard shortcut SHIFT + CTRL + / (Toggle block comment)
  • File chooser dialog for Google Workspace private key file import in configuration

v1.4.58 (2020-08-07)

  • API throughput optimized
  • Elasticsearch - retrieving numeric fields from _source when represented as strings
  • UI, API - do not emit extra : when serializing IPv6 addresses ending with 0 segment
  • Map - avoid showing popup message when selecting data range with no features

v1.4.57 (2020-07-17)

  • Increased Map maximum zoom level by 4
  • Preview/Prepare Query/Prepare Pattern/Download buttons are enabled for files of any size in Data Browser
  • Queries fail to start on systems with unavailable native zlib support

v1.4.56 (2020-06-29)

  • Raw cursor for cached files over 2GB in size causes query to fail
  • JDBC - IP*, MACADDR values garbled when inserted into MySQL database

v1.4.55 (2020-06-19)

  • Autocomplete does not work in Edge browser

v1.4.54 (2020-06-11)

  • Support for not following symbolic links for file:// datastores.
  • Shared Access Token as an authentication method for wasb:// datastores (Microsoft Azure Blob Storage).
  • Support for reading and writing bytes type via JDBC (MySQL LONGBLOB/BLOB, PostgreSQL bytea, Oracle BLOB, SQLite BLOB)
  • Chart supports logarithmic y-axis scale (select via menu)
  • Timestamp parsing format letters Z and z support timezones like GMT+3, GMT-20
  • Resultset browser columns can be selected and hidden
  • Elasticsearch access functionality is not present in SpectX Desktop Edition
  • Linux startup script in “enable-boot-start” mode fails to start SpectX when run as root for different non-root user
  • First row in resultset browser is not selectable

v1.4.53 (2020-04-24)

  • bin\spectx.bat and bin\spectx.env.bat.default removed from Windows distribution of the SpectX in favor of newly introduced native service launcher (see below).
  • JRE upgraded to 1.8.0_252 in packages for all platforms.
  • Added: Mandatory EULA acceptance. If EULA, shipped with the product, has not been accepted yet, it gets presented for acceptance to an user with administrator privileges on the user’s login after startup.
  • Service launcher to SpectX server for Windows.
  • Automatic detection and support for big-endian PCAP files via PCAP(bigEndian=true)
  • Confirmation dialog when closing browser tab/window
  • Option to freeze y-axis in chart menu
  • Support for coloring resultset rows and columns
  • Chart title is visible on .png and .svg exports
  • Chart title is editable
  • Chart selection filter in case of descending x-axis
  • “Display rows as” functionality for CSV, TSV, TTSV formats have extraneous header column
  • Saving query resultset to a nonexistent directory
  • Saving query resultset can overwrite existing file

v1.4.52 (2020-02-29)

  • SpectXDesktop uses integrated GPU on OSX
  • Script files can not be opened after setting user timezone to PST

v1.4.51 (2020-02-25)

  • Option to hide inaccessible folders in resource tree, configurable per user
  • CTRL + double click, CTRL + ENTER on file resource in resource tree opens it in new tab even if one is already open
  • ‘U’ TIMESTAMP parser format letter for unnecessary non-numeric chars
  • Configuration setting engine.da.http.user-agent
  • Added left join query processing command
  • LIMIT() following PARSE() does not accept numbers bigger than 0x7fffffffffffffff
  • PARSE() does not accept NULL as value for pattern, archive_src and rc parameters

v1.4.47 (2020-01-29)

  • JRE upgraded to 1.8.0_242 (Windows64, Linux64, OSX64) and 1.8.0_232 (Arch Linux ARM 64)

v1.4.46 (2020-01-14)


v1.4.44 (2019-12-10)

  • Chart PNG export at double scale for better quality
  • Support for Elasticsearch 7.x
  • Chart x-axis labels are masked on left and right edges

v1.4.41 (2019-12-03)

  • Datastore creation and configuration flow in WGUI:
  • Reordered buttons in dialogs such that primary button is on the left.
  • Removed ‘Log File’, ‘Data File’ and ‘Text File’ from the ‘New’ menu and resource tree context menus.
  • Double click on a datastore in the resource tree opens the data browser, ‘Configure’ in context menu opens configuration
  • ES and SAVE_ES have and query.save_es.proxy options with fallback to http(s).proxyHost and http(s).proxyPort system properties
  • IPNET parser
  • API fails to serve empty resultset.
  • WGUI unintended session terminations.

v1.4.40 (2019-11-25)

  • DataStore Editor: “Browse” functionality retired in favor of simple connectivity test:
    • “Browse” button renamed to “Test Connectivity”, which when pushed performs an attempt to connect to current datastore using realtime values of configuration parameters and list files in its root directory.
    • The “Test Connectivity” button is disabled for dislocated datastores.
  • Improved responsiveness of map visualization with a large number of features
  • LOAD and LOAD_HTTP UDF builtin functions removed in favor of a new FETCH function
  • SpectX configuration file:
    • ${PROP} construction in configuration file variables, refers corresponding environment variable first, and if it is undefined, refers then corresponding system property
    • configuration parameters expecting directory paths do not support empty values
    • log directory gets created if it is specified but does not exist
  • TIMESTAMP_NANO matcher
  • ‘f’ TIMESTAMP parser format letter for fractional seconds
  • An existing pattern file can be dragged from resource tree to an open pattern editor
  • Resultset column layout is retained across query executions.
  • Datastore ACL retains comments
  • Saved patterns in subfolders are also considered for Input Data Browser -> prepare pattern
  • SpectX API: capability to retrieve the schema of resultsets (for tables, queries)
  • File name filter in DataBrowser’s data store view
  • Resultset column selection is not vertically scrollable.
  • Map initial focus fails for features with invalid coordinates. Now Map supports features with invalid coordinates (clamped to max/min values)

v1.4.39 (2019-09-11)

  • Min password length changed from 5 to 8 characters
  • Order of searching of datastores in resource tree changed from system, shared, user to user, shared, system.
  • UI: System Status > Running Queries: sorted state gets lost after a second
  • ISO8601 Timestamp Format Matcher does not honor RFC
  • Globbed blob read ACLs with exact match for gzip files in SA/SAS datastores deny processing of these files if these get indexed on SourceAgent side and get cached on SpectX side.

v1.4.38 (2019-08-27)

  • Support for reading data from 7z and Rar (v4 and below) archives. See more at Working With Archives.
  • DataAccess/S3: bucket listing response version mismatch handling

v1.4.37 (2019-07-05)

  • SpectX UI: last login time to users table in Users tab under “Admin - Users and Groups” menu
  • Possibility to configure startup at boot in Linux (with Systemd/SysV-style init/Upstart init systems)
  • Filter(ES_QUERY()) support for Elasticsearch 5.6.*
  • Linux/OSX startup script renamed to . Helper scripts removed.

v1.4.36 (2019-06-13)

  • IWA authenticator: configurable LDAP group name transformation and possibility to abandon group membership information retrieval with user information LDAP query.
  • IWA authenticator: fallback to basic authentication which allows users from machines not joined to Active Directory domain to log in to SpectX with their domain username and password
  • SpectX default environment setup script for linux/osx platforms sets umask to 0077.
  • JRE-bundled packages with SpectX for Arch Linux ARM 64bit (aarch64)
  • UI: Chart supports multiple x-axes to visualize two sets of superimposed value ranges
  • Parsing JSON objects now allows exporting selected members directly to resultset columns using JSON{…}(flat=true) config parameter
  • User Defined Functions can now include multiple statements.
  • New functions: GZIP(), GUNZIP()
  • Improved listing operations optimisation in cloud (Amazon/Azure/Google) data stores.
  • Blobs disallowed from reading by blob ACL are not displayed in data browser and get not included in listing result by default; new settings for both data browser and listing can be used to configure the behavior correspondingly
  • Globbed blob read ACLs with exact match for gzip files in SA/SAS datastores deny processing of these files if these get indexed on SourceAgent side.
  • In Firefox 67+ submenus of menu ‘Save’ > ‘Result As’ do not work
  • Chart legend does not respond to resize when many columns are displayed

v1.4.35 (2019-04-23)

  • IWA authenticator: support for multiple search bases in user and group LDAP query filter expressions
  • Elasticsearch support: indexes & structure listing, native search using Query String
  • DataAccess/S3: support for path-style access to buckets
  • Elasticsearch support: enhanced control over large responses from Elasticsearch, optimized search predicate pushdown by object subfields
  • UI: Incorrect group list rendering in user properties window in case of list sizes greater than 40.

v1.4.34 (2019-04-01)

  • Elasticsearch support: connectivity issues to 6.6+ clusters

v1.4.33 (2019-03-29)

  • UI:
    • Update download progress indication and cancelling possibility in SpectX Update dialog.
    • Data Store editor: glob patterns can be used in data store ACL definitions, in addition to prefixed uri paths.
    • Resource tree filter. Defaults to filename filter, c:<string> searches for files containing ‘string’, and t:<hour/day/week> searches for files modified last hour/day/week.
    • Result set _raw_text column contents exportable to file from ‘Save’ and right click context menus
    • Query filter creation from resultset selection supports time ranges for timestamp columns
  • Support for reading data from ZIP archives. See more at Working With Archives.
  • UserAdmin role.
  • Engine:
    • Increased processing speed of Bzip2 files on Intel-based 64 bit OSX and Linux platforms by 20%.
  • UI:
    • Improved editor auto-completion in nested function calls
    • Data Store Editor: ‘.’ symbol is allowed in store name
  • UI:
    • Map feature properties popup displays correct values for tuple type columns
    • Export of chart with column names containing non-ASCII characters

v1.4.31 (2019-02-22)

  • SpectX configuration keys engine.fs_access and engine.fs_unmanaged_access are deprecated. Instead, new configuration keys engine.da.protocol.<protocol> can be used to define usage permissions for each data access protocol separately.

v1.4.30 (2019-02-11)

  • DataAccess/HDFS: High availability clusters accessibility support

v1.4.29 (2019-02-08)

  • OpenJDK 8 HotSpot Java Runtime Environment is now supported
  • DataAccess/HDFS: improved accessing secure HDFS clusters with multiple Kerberos authentication realms configured
  • On Windows, Data Browser started by pressing Browse button in data store configuration editor window for previously stored configuration, fails to connect to the data store.

v1.4.28 (2019-01-10)

This release contains bug fixes, a few new features and some improvements.

  • Reading data from relational databases: refactored input stream commands
  • UI:
    • Data Store editor: the browse-button uses real-time values in the editor form to connect to the data store. Browsing is restricted to the currently editable store only.
    • DataBrowser: keyboard navigation improved. Arrow, Pg Down/Up, Enter and Backspace keys can be used for navigation in all views.
    • “Admin - Users and Groups” menu: added descriptive text to the User/Group/ExtGroupMapping deletion dialogs.
    • “System status” menu: remastered Effective Settings view.
    • Script and Pattern editors: removed ctrl - shift - arrow tab navigation, text of displayed error messages made selectable.
    • The full name of a logged in user (if set) is used instead of a username in the rightmost menu at the top bar
    • Background version checker: connection exception logging with debug log level (previously: error log level).
  • Engine:
    • Improved processing performance of concatenated GZ/ZZ/PiGZ/PiZZ files.
    • Increased processing speed of GZ/ZZ/PiGZ/PiZZ files on Intel-based 64 bit OSX and Linux platforms by 20%.
  • Listing queries for cloud (Amazon/Azure/Google cloud) targets with the ‘includeContainers’:true argument.
  • Removed unnecessary creation of the folder “users” under the path specified by the config parameter sx.user_data.dir.
  • Failure to attempt to create own log directory on the very first run fixed in both SpectX and SourceAgent startup Windows scripts.