Release Notes

v1.4.63 (2020-09-17)

  • Negative indexes support for SUBSTR function.
  • Export result as CCSV produces a file with .csv extension
  • Support for offset and length arguments in Data Access URI Syntax’s fragment part and compress_type field in the LIST have been removed. Instead, newly introduced slicing and decompression instructions can be used.
  • Resource tree expands too many folders on navigation
  • API - free result table when cancelling query
  • File rename dialog layout garbled for long filenames
  • ‘Display selection as’ with no rows selected displays 0 rows
  • Resource API access tab - limit response preview output size to avoid freezing UI

v1.4.61 (2020-08-24)

  • Copy/Paste functions for files and folders to right-click item menus in Resource Tree.
  • Listing of sx:/ datastore with URIs having glob patterns in directory names of the path returns incomplete results (regression bug since v1.4.57)

v1.4.60 (2020-08-19)

  • Map - do not constrain zoom when features are close together
  • Resource API access tab - response preview
  • SourceAgent sa/sas protocols listing optimisation regression (since v1.4.55)

v1.4.59 (2020-08-14)

  • Markdown table export format
  • Ndjson export format
  • New keyboard shortcut CTRL + / (Toggle line comment)
  • New keyboard shortcut SHIFT + CTRL + / (Toggle block comment)
  • File chooser dialog for G Suite private key file import in configuration

v1.4.58 (2020-08-07)

  • API throughput optimized
  • Elasticsearch - retrieving numeric fields from _source when represented as strings
  • UI, API - do not emit extra : when serializing IPv6 addresses ending with 0 segment
  • Map - avoid showing popup message when selecting data range with no features

v1.4.57 (2020-07-17)

  • Increased Map maximum zoom level by 4
  • Preview/Prepare Query/Prepare Pattern/Download buttons are enabled for files of any size in Data Browser
  • Queries fail to start on systems with unavailable native zlib support

v1.4.56 (2020-06-29)

  • Raw cursor for cached files over 2GB in size causes query to fail
  • JDBC - IP*, MACADDR values garbled when inserted into MySQL database

v1.4.55 (2020-06-19)

  • Autocomplete does not work in Edge browser

v1.4.54 (2020-06-11)

  • Support for not following symbolic links for file:// datastores.
  • Shared Access Token as an authentication method for wasb:// datastores (Microsoft Azure Blob Storage).
  • Support for reading and writing bytes type via JDBC (MySQL LONGBLOB/BLOB, PostgreSQL bytea, Oracle BLOB, SQLite BLOB)
  • Chart supports logarithmic y-axis scale (select via menu)
  • Timestamp parsing format letters Z and z support timezones like GMT+3, GMT-20
  • Resultset browser columns can be selected and hidden
  • Elasticsearch access functionality is not present in SpectX Desktop Edition
  • Linux startup script in “enable-boot-start” mode fails to start SpectX when run as root for different non-root user
  • First row in resultset browser is not selectable

v1.4.53 (2020-04-24)

  • bin\spectx.bat and bin\spectx.env.bat.default removed from Windows distribution of the SpectX in favor of newly introduced native service launcher (see below).
  • JRE upgraded to 1.8.0_252 in packages for all platforms.
  • Added: Mandatory EULA acceptance. If EULA, shipped with the product, has not been accepted yet, it gets presented for acceptance to an user with administrator privileges on the user’s login after startup.
  • Service launcher to SpectX server for Windows.
  • Automatic detection and support for big-endian PCAP files via PCAP(bigEndian=true)
  • Confirmation dialog when closing browser tab/window
  • Option to freeze y-axis in chart menu
  • Support for coloring resultset rows and columns
  • Chart title is visible on .png and .svg exports
  • Chart title is editable
  • Chart selection filter in case of descending x-axis
  • “Display rows as” functionality for CSV, TSV, TTSV formats have extraneous header column
  • Saving query resultset to a nonexistent directory
  • Saving query resultset can overwrite existing file

v1.4.52 (2020-02-29)

  • SpectXDesktop uses integrated GPU on OSX
  • Script files can not be opened after setting user timezone to PST

v1.4.51 (2020-02-25)

  • Option to hide inaccessible folders in resource tree, configurable per user
  • CTRL + double click, CTRL + ENTER on file resource in resource tree opens it in new tab even if one is already open
  • ‘U’ TIMESTAMP parser format letter for unnecessary non-numeric chars
  • Configuration setting engine.da.http.user-agent
  • Added left join query processing command
  • LIMIT() following PARSE() does not accept numbers bigger than 0x7fffffffffffffff
  • PARSE() does not accept NULL as value for pattern, archive_src and rc parameters

v1.4.47 (2020-01-29)

  • JRE upgraded to 1.8.0_242 (Windows64, Linux64, OSX64) and 1.8.0_232 (Arch Linux ARM 64)

v1.4.46 (2020-01-14)


v1.4.44 (2019-12-10)

  • Chart PNG export at double scale for better quality
  • Support for ElasticSearch 7.x
  • Chart x-axis labels are masked on left and right edges

v1.4.41 (2019-12-03)

  • Datastore creation and configuration flow in WGUI:
  • Reordered buttons in dialogs such that primary button is on the left.
  • Removed ‘Log File’, ‘Data File’ and ‘Text File’ from the ‘New’ menu and resource tree context menus.
  • Double click on a datastore in the resource tree opens the data browser, ‘Configure’ in context menu opens configuration
  • ES and SAVE_ES have and query.save_es.proxy options with fallback to http(s).proxyHost and http(s).proxyPort system properties
  • IPNET parser
  • API fails to serve empty resultset.
  • WGUI unintended session terminations.

v1.4.40 (2019-11-25)

  • DataStore Editor: “Browse” functionality retired in favor of simple connectivity test:
    • “Browse” button renamed to “Test Connectivity”, which when pushed performs an attempt to connect to current datastore using realtime values of configuration parameters and list files in its root directory.
    • The “Test Connectivity” button is disabled for dislocated datastores.
  • Improved responsiveness of map visualization with a large number of features
  • LOAD and LOAD_HTTP UDF builtin functions removed in favor of a new FETCH function
  • SpectX configuration file:
    • ${PROP} construction in configuration file variables, refers corresponding environment variable first, and if it is undefined, refers then corresponding system property
    • configuration parameters expecting directory paths do not support empty values
    • log directory gets created if it is specified but does not exist
  • TIMESTAMP_NANO matcher
  • ‘f’ TIMESTAMP parser format letter for fractional seconds
  • An existing pattern file can be dragged from resource tree to an open pattern editor
  • Resultset column layout is retained across query executions.
  • Datastore Read ACL retains comments
  • Saved patterns in subfolders are also considered for Input Data Browser -> prepare pattern
  • SpectX API: capability to retrieve the schema of resultsets (for tables, queries)
  • File name filter in DataBrowser’s data store view
  • Resultset column selection is not vertically scrollable.
  • Map initial focus fails for features with invalid coordinates. Now Map supports features with invalid coordinates (clamped to max/min values)

v1.4.39 (2019-09-11)

  • Min password length changed from 5 to 8 characters
  • Order of searching of datastores in resource tree changed from system, shared, user to user, shared, system.
  • UI: System Status > Running Queries: sorted state gets lost after a second
  • ISO8601 Timestamp Format Matcher does not honor RFC
  • Globbed blob read ACLs with exact match for gzip files in SA/SAS datastores deny processing of these files if these get indexed on SourceAgent side and get cached on SpectX side.

v1.4.38 (2019-08-27)

  • Support for reading data from 7z and Rar (v4 and below) archives. See more at Working With Archives.
  • DataAccess/S3: bucket listing response version mismatch handling

v1.4.37 (2019-07-05)

  • SpectX UI: last login time to users table in Users tab under “Admin - Users and Groups” menu
  • Possibility to configure startup at boot in Linux (with Systemd/SysV-style init/Upstart init systems)
  • Filter(ES_QUERY()) support for ElasticSearch 5.6.*
  • Linux/OSX startup script renamed to . Helper scripts removed.

v1.4.36 (2019-06-13)

  • IWA authenticator: configurable LDAP group name transformation and possibility to abandon group membership information retrieval with user information LDAP query.
  • IWA authenticator: fallback to basic authentication which allows users from machines not joined to Active Directory domain to log in to SpectX with their domain username and password
  • SpectX default environment setup script for linux/osx platforms sets umask to 0077.
  • JRE-bundled packages with SpectX for Arch Linux ARM 64bit (aarch64)
  • UI: Chart supports multiple x-axes to visualize two sets of superimposed value ranges
  • Parsing JSON objects now allows exporting selected members directly to resultset columns using JSON{…}(flat=true) config parameter
  • User Defined Functions can now include multiple statements.
  • New functions: GZIP(), GUNZIP()
  • Improved listing operations optimisation in cloud (Amazon/Azure/Google) data stores.
  • Blobs disallowed from reading by blob Read ACL are not displayed in data browser and get not included in listing result by default; new settings for both data browser and listing can be used to configure the behavior correspondingly
  • Globbed blob read ACLs with exact match for gzip files in SA/SAS datastores deny processing of these files if these get indexed on SourceAgent side.
  • In Firefox 67+ submenus of menu ‘Save’ > ‘Result As’ do not work
  • Chart legend does not respond to resize when many columns are displayed

v1.4.35 (2019-04-23)

  • IWA authenticator: support for multiple search bases in user and group LDAP query filter expressions
  • Elasticsearch support: indexes & structure listing, native search using Query String
  • DataAccess/S3: support for path-style access to buckets
  • Elasticsearch support: enhanced control over large responses from Elasticsearch, optimized search predicate delegation by object subfields
  • UI: Incorrect group list rendering in user properties window in case of list sizes greater than 40.

v1.4.34 (2019-04-01)

  • Elasticsearch support: connectivity issues to 6.6+ clusters

v1.4.33 (2019-03-29)

  • UI:
    • Update download progress indication and cancelling possibility in SpectX Update dialog.
    • Data Store editor: glob patterns can be used in data store Read ACL definitions, in addition to prefixed uri paths.
    • Resource tree filter. Defaults to filename filter, c:<string> searches for files containing ‘string’, and t:<hour/day/week> searches for files modified last hour/day/week.
    • Result set _raw_text column contents exportable to file from ‘Save’ and right click context menus
    • Query filter creation from resultset selection supports time ranges for timestamp columns
  • Support for reading data from ZIP archives. See more at Working With Archives.
  • UserAdmin role.
  • Engine:
    • Increased processing speed of Bzip2 files on Intel-based 64 bit OSX and Linux platforms by 20%.
  • UI:
    • Improved editor auto-completion in nested function calls
    • Data Store Editor: ‘.’ symbol is allowed in store name
  • UI:
    • Map feature properties popup displays correct values for tuple type columns
    • Export of chart with column names containing non-ASCII characters

v1.4.31 (2019-02-22)

  • SpectX configuration keys engine.fs_access and engine.fs_unmanaged_access are deprecated. Instead, new configuration keys engine.da.protocol.<protocol> can be used to define usage permissions for each data access protocol separately.

v1.4.30 (2019-02-11)

  • DataAccess/HDFS: High availability clusters accessibility support

v1.4.29 (2019-02-08)

  • OpenJDK 8 HotSpot Java Runtime Environment is now supported
  • DataAccess/HDFS: improved accessing secure HDFS clusters with multiple Kerberos authentication realms configured
  • On Windows, Data Browser started by pressing Browse button in data store configuration editor window for previously stored configuration, fails to connect to the data store.

v1.4.28 (2019-01-10)

This release contains bug fixes, a few new features and some improvements.

  • Reading data from relational databases: refactored input stream commands
  • UI:
    • Data Store editor: the browse-button uses real-time values in the editor form to connect to the data store. Browsing is restricted to the currently editable store only.
    • DataBrowser: keyboard navigation improved. Arrow, Pg Down/Up, Enter and Backspace keys can be used for navigation in all views.
    • “Admin - Users and Groups” menu: added descriptive text to the User/Group/ExtGroupMapping deletion dialogs.
    • “System status” menu: remastered Effective Settings view.
    • Script and Pattern editors: removed ctrl - shift - arrow tab navigation, text of displayed error messages made selectable.
    • The full name of a logged in user (if set) is used instead of a username in the rightmost menu at the top bar
    • Background version checker: connection exception logging with debug log level (previously: error log level).
  • Engine:
    • Improved processing performance of concatenated GZ/ZZ/PiGZ/PiZZ files.
    • Increased processing speed of GZ/ZZ/PiGZ/PiZZ files on Intel-based 64 bit OSX and Linux platforms by 20%.
  • Listing queries for cloud (Amazon/Azure/Google cloud) targets with the ‘includeContainers’:true argument.
  • Removed unnecessary creation of the folder “users” under the path specified by the config parameter sx.user_data.dir.
  • Failure to attempt to create own log directory on the very first run fixed in both SpectX and SourceAgent startup Windows scripts.