Release Notes

v1.4.68 (2020-11-27)

Changed:
  • “All platforms (no JRE)” package of SpectX Server Edition does not contain “tools” directory with SourceAgent and SxGzip Compression Utility anymore, as these both are available as separate downloads, in both licensee cabinet and SpectX products download page.
  • msapi:// and gsuite:// protocols availability in SpectX with Free license is extended until 1st February 2021.
  • JRE upgraded to 1.8.0_275 in packages for all platforms.
Added:
  • Config key wgui.instanceName for displaying configured value under SpectX logo on login page and in main view. Available only in Server Edition.
  • Configuration properties can be defined as environment variables instead of a configuration file or to override certain properties in the configuration file.
  • Editor autocomplete - LIST and PARSE input and process command parameters and configuration settings.
  • Editor autocomplete - stream & expression blocks support:
    • when inside block declaration, complete variables declared in enclosing scopes & parameters of enclosing blocks.
    • for block invocations, complete parameter names.
  • Editor autocomplete - show descriptions for pipe task fields.
  • CREATE FUNCTION JavaScript directive can return VARIANT_OBJECT type.
  • Configurable list of Java classes exposed to CREATE FUNCTION JavaScript directive via engine.js.exposed_classes.
  • Color of a resultset field named xyz can be specified via column _color_xyz value.
Fixed:
  • In Configuration window of the SpectX Desktop, cache directory path obtained by means of a file chooser dialog is not saved in the configuration.
  • On linux, SpectX update via UI fails if SpectX is launched with systemd.
  • ElasticSearch compatibility for versions 8x, <2.0.0, <5.0.0, 5.1.2 - 5.3.2.
  • Editor autocomplete - ES and PARSE commands parameters do not complete when command name is expressed in mixed case.
  • Resultset browser error reporting improved when data files have been manually removed.
  • LIST process command - parameters specified in source stream do not take effect when also selected in output.
  • SpectXDesktop sometimes crashes on OSX when its window is moved from main screen to external monitor screen.

v1.4.67 (2020-11-03)

Changed:
  • “Change License” sub-menu moved from “Maintenance” menu to “About”
  • LIST() input command - accepts parameters when src is a STRING.
Added:
  • Resultset Code Snippet menu - time distribution for TIMESTAMP_NANO columns.
  • Docker image features to support no-interaction deployment:
    • -E/–eula argument shows EULA.
    • SPECTX_ACCEPT_EULA=true environment variable accepts EULA.
    • SPECTX_ADMIN_PASSWORD environment variable specifies password for admin user that doesn’t have to be changed on first login.
  • New UDF function ARRAY_REVERSE.
Fixed:
  • msapi:// and gsuite:// protocols produce an excessive virtual file for overlap hour occurring in transition from DST when datastore is configured to have hour pointer in both file name and path.
  • Configuration key wgui.dataBrowser.showBlobsDisallowedInACL is ignored by data browser.
  • SpectX may crash on Linux if launched with system (not bundled) JRE when processing gzip or bzip2 compressed files.
  • LIST() - tz parameter in src tuple ignored.
  • DUAL() result tuple[ip6] and tuple[ip6s] contain ipv4 addresses.
  • Some keywords were not highlighted in script editor.
  • Data store with execute but no read permission cannot be opened from resource tree.

v1.4.66 (2020-10-15)

Changed:
Added:
  • LIKE supports matching a single character with “_”.
Fixed:
  • Resource api access view without configured token
  • Renaming resource while changing only filename case deletes it
  • Processing of some concatenated gzip files fails on Intel-based 64 bit OSX and Linux platforms.

v1.4.65 (2020-10-05)

Added:
  • SAVE_ES() - support writing column types:
    • ARRAY, VARIANT_ARRAY -> multivalued field
    • TUPLE, VARIANT_OBJECT, MAP -> Object
    • DURATION-> Long
    • IPNET -> Ip_range
    • TIMESTAMP_NANO -> Date
    • IPSOCKET -> Text
    • MACADDR -> Text
  • ES() - support reading index types:
    • Ip_range -> IPNET
    • Date_nanos -> TIMESTAMP_NANO
  • msapi:// protocol for accessing Azure Active Directory, Office 365 and Azure activity audit logs through Azure Active Directory Activity reports, Office 365 Management Activity and Azure Monitoring REST APIs
  • Editor autocomplete - positional call parameters for $() @() @@()
  • Editor autocomplete - show default values for resource stream init parameters
Fixed:
  • SAVE_ES() - ElasticSearch connections are not closed
  • ES() - retrieving string fields from _source when represented as non-strings
  • Resultset browser - HTML injection in column headers
  • API - parallel serialization misbehaves with sort()

v1.4.64 (2020-09-23)

Added:
  • Editor autocomplete - suggest resource stream call parameters
  • Editor autocomplete - suggested resources have tooltip with path link
  • New pipe command select_replace
  • New pipe command select_rename
  • New pipe command select_remove
  • Allows extra comma at the end of list. For example | select(field, field2, field3,)
Fixed:
  • WindowsDefender exclusion paths querying fails starting with Windows 10 version 2004

v1.4.63 (2020-09-17)

Changed:
  • Negative indexes support for SUBSTR function.
  • Export result as CCSV produces a file with .csv extension
  • Support for offset and length arguments in Data Access URI Syntax’s fragment part and compress_type field in the LIST have been removed. Instead, newly introduced slicing and decompression instructions can be used.
Added:
Fixed:
  • Resource tree expands too many folders on navigation
  • API - free result table when cancelling query
  • File rename dialog layout garbled for long filenames
  • ‘Display selection as’ with no rows selected displays 0 rows
  • Resource API access tab - limit response preview output size to avoid freezing UI

v1.4.61 (2020-08-24)

Added:
  • Copy/Paste functions for files and folders to right-click item menus in Resource Tree.
Fixed:
  • Listing of sx:/ datastore with URIs having glob patterns in directory names of the path returns incomplete results (regression bug since v1.4.57)

v1.4.60 (2020-08-19)

Changed:
  • Map - do not constrain zoom when features are close together
Added:
  • Resource API access tab - response preview
Fixed:
  • SourceAgent sa/sas protocols listing optimisation regression (since v1.4.55)

v1.4.59 (2020-08-14)

Added:
  • Markdown table export format
  • Ndjson export format
  • New keyboard shortcut CTRL + / (Toggle line comment)
  • New keyboard shortcut SHIFT + CTRL + / (Toggle block comment)
  • File chooser dialog for G Suite private key file import in configuration

v1.4.58 (2020-08-07)

Changed:
  • API throughput optimized
Added:
Fixed:
  • Elasticsearch - retrieving numeric fields from _source when represented as strings
  • UI, API - do not emit extra : when serializing IPv6 addresses ending with 0 segment
  • Map - avoid showing popup message when selecting data range with no features

v1.4.57 (2020-07-17)

Changed:
  • Increased Map maximum zoom level by 4
  • Preview/Prepare Query/Prepare Pattern/Download buttons are enabled for files of any size in Data Browser
Fixed:
  • Queries fail to start on systems with unavailable native zlib support

v1.4.56 (2020-06-29)

Added:
Fixed:
  • Raw cursor for cached files over 2GB in size causes query to fail
  • JDBC - IP*, MACADDR values garbled when inserted into MySQL database

v1.4.55 (2020-06-19)

Fixed:
  • Autocomplete does not work in Edge browser

v1.4.54 (2020-06-11)

Added:
  • Support for not following symbolic links for file:// datastores.
  • Shared Access Token as an authentication method for wasb:// datastores (Microsoft Azure Blob Storage).
  • Support for reading and writing bytes type via JDBC (MySQL LONGBLOB/BLOB, PostgreSQL bytea, Oracle BLOB, SQLite BLOB)
  • Chart supports logarithmic y-axis scale (select via menu)
  • Timestamp parsing format letters Z and z support timezones like GMT+3, GMT-20
  • Resultset browser columns can be selected and hidden
Fixed:
  • Elasticsearch access functionality is not present in SpectX Desktop Edition
  • Linux startup script in “enable-boot-start” mode fails to start SpectX when run as root for different non-root user
  • First row in resultset browser is not selectable

v1.4.53 (2020-04-24)

Changed:
  • bin\spectx.bat and bin\spectx.env.bat.default removed from Windows distribution of the SpectX in favor of newly introduced native service launcher (see below).
  • JRE upgraded to 1.8.0_252 in packages for all platforms.
Added:
  • Added: Mandatory EULA acceptance. If EULA, shipped with the product, has not been accepted yet, it gets presented for acceptance to an user with administrator privileges on the user’s login after startup.
  • Service launcher to SpectX server for Windows.
  • Automatic detection and support for big-endian PCAP files via PCAP(bigEndian=true)
  • Confirmation dialog when closing browser tab/window
  • Option to freeze y-axis in chart menu
  • Support for coloring resultset rows and columns
  • Chart title is visible on .png and .svg exports
  • Chart title is editable
Fixed:
  • Chart selection filter in case of descending x-axis
  • “Display rows as” functionality for CSV, TSV, TTSV formats have extraneous header column
  • Saving query resultset to a nonexistent directory
  • Saving query resultset can overwrite existing file

v1.4.52 (2020-02-29)

Changed:
  • SpectXDesktop uses integrated GPU on OSX
Fixed:
  • Script files can not be opened after setting user timezone to PST

v1.4.51 (2020-02-25)

Added:
  • Option to hide inaccessible folders in resource tree, configurable per user
  • CTRL + double click, CTRL + ENTER on file resource in resource tree opens it in new tab even if one is already open
  • Operators IS TRUE, IS FALSE, IS NOT TRUE, IS NOT FALSE
  • ‘U’ TIMESTAMP parser format letter for unnecessary non-numeric chars
  • Configuration setting engine.da.http.user-agent
  • Added left join query processing command
Fixed:
  • LIMIT() following PARSE() does not accept numbers bigger than 0x7fffffffffffffff
  • PARSE() does not accept NULL as value for pattern, archive_src and rc parameters

v1.4.47 (2020-01-29)

Changed:
  • JRE upgraded to 1.8.0_242 (Windows64, Linux64, OSX64) and 1.8.0_232 (Arch Linux ARM 64)
Added:

v1.4.46 (2020-01-14)

Added:

v1.4.44 (2019-12-10)

Changed:
  • Chart PNG export at double scale for better quality
Added:
  • Support for ElasticSearch 7.x
Fixed:
  • Chart x-axis labels are masked on left and right edges

v1.4.41 (2019-12-03)

Changed:
  • Datastore creation and configuration flow in WGUI:
  • Reordered buttons in dialogs such that primary button is on the left.
  • Removed ‘Log File’, ‘Data File’ and ‘Text File’ from the ‘New’ menu and resource tree context menus.
  • Double click on a datastore in the resource tree opens the data browser, ‘Configure’ in context menu opens configuration
Added:
  • ES and SAVE_ES have query.es.proxy and query.save_es.proxy options with fallback to http(s).proxyHost and http(s).proxyPort system properties
  • IPNET parser
  • new UDF functions ARRAY_FIRST, ARRAY_LAST, ARRAY_UNIQ
Fixed:
  • API fails to serve empty resultset.
  • WGUI unintended session terminations.

v1.4.40 (2019-11-25)

Changed:
  • DataStore Editor: “Browse” functionality retired in favor of simple connectivity test:
    • “Browse” button renamed to “Test Connectivity”, which when pushed performs an attempt to connect to current datastore using realtime values of configuration parameters and list files in its root directory.
    • The “Test Connectivity” button is disabled for dislocated datastores.
  • Improved responsiveness of map visualization with a large number of features
  • LOAD and LOAD_HTTP UDF builtin functions removed in favor of a new FETCH function
  • SpectX configuration file:
    • ${PROP} construction in configuration file variables, refers corresponding environment variable first, and if it is undefined, refers then corresponding system property
    • configuration parameters expecting directory paths do not support empty values
    • log directory gets created if it is specified but does not exist
Added:
  • TIMESTAMP_NANO matcher
  • ‘f’ TIMESTAMP parser format letter for fractional seconds
  • An existing pattern file can be dragged from resource tree to an open pattern editor
  • Resultset column layout is retained across query executions.
  • Datastore Read ACL retains comments
  • Saved patterns in subfolders are also considered for Input Data Browser -> prepare pattern
  • SpectX API: capability to retrieve the schema of resultsets (for tables, queries)
  • File name filter in DataBrowser’s data store view
Fixed:
  • Resultset column selection is not vertically scrollable.
  • Map initial focus fails for features with invalid coordinates. Now Map supports features with invalid coordinates (clamped to max/min values)

v1.4.39 (2019-09-11)

Changed:
  • Min password length changed from 5 to 8 characters
  • Order of searching of datastores in resource tree changed from system, shared, user to user, shared, system.
Fixed:
  • UI: System Status > Running Queries: sorted state gets lost after a second
  • ISO8601 Timestamp Format Matcher does not honor RFC
  • Globbed blob read ACLs with exact match for gzip files in SA/SAS datastores deny processing of these files if these get indexed on SourceAgent side and get cached on SpectX side.

v1.4.38 (2019-08-27)

Added:
  • Support for reading data from 7z and Rar (v4 and below) archives. See more at Working With Archives.
  • DataAccess/S3: bucket listing response version mismatch handling

v1.4.37 (2019-07-05)

Added:
  • SpectX UI: last login time to users table in Users tab under “Admin - Users and Groups” menu
  • Possibility to configure startup at boot in Linux (with Systemd/SysV-style init/Upstart init systems)
  • Filter(ES_QUERY()) support for ElasticSearch 5.6.*
Changed:
  • Linux/OSX startup script spectx-init.d.sh renamed to spectx.sh . Helper scripts spectx.common.sh removed.

v1.4.36 (2019-06-13)

Added:
  • IWA authenticator: configurable LDAP group name transformation and possibility to abandon group membership information retrieval with user information LDAP query.
  • IWA authenticator: fallback to basic authentication which allows users from machines not joined to Active Directory domain to log in to SpectX with their domain username and password
  • SpectX default environment setup script for linux/osx platforms sets umask to 0077.
  • JRE-bundled packages with SpectX for Arch Linux ARM 64bit (aarch64)
  • UI: Chart supports multiple x-axes to visualize two sets of superimposed value ranges
  • Parsing JSON objects now allows exporting selected members directly to resultset columns using JSON{…}(flat=true) config parameter
  • User Defined Functions can now include multiple statements.
  • New functions: GZIP(), GUNZIP()
Changed:
  • Improved listing operations optimisation in cloud (Amazon/Azure/Google) data stores.
  • Blobs disallowed from reading by blob Read ACL are not displayed in data browser and get not included in listing result by default; new settings for both data browser and listing can be used to configure the behavior correspondingly
Fixed:
  • Globbed blob read ACLs with exact match for gzip files in SA/SAS datastores deny processing of these files if these get indexed on SourceAgent side.
  • In Firefox 67+ submenus of menu ‘Save’ > ‘Result As’ do not work
  • Chart legend does not respond to resize when many columns are displayed

v1.4.35 (2019-04-23)

Added:
  • IWA authenticator: support for multiple search bases in user and group LDAP query filter expressions
  • Elasticsearch support: indexes & structure listing, native search using Query String
  • DataAccess/S3: support for path-style access to buckets
Changed:
  • Elasticsearch support: enhanced control over large responses from Elasticsearch, optimized search predicate delegation by object subfields
Fixed:
  • UI: Incorrect group list rendering in user properties window in case of list sizes greater than 40.

v1.4.34 (2019-04-01)

Fixed:
  • Elasticsearch support: connectivity issues to 6.6+ clusters

v1.4.33 (2019-03-29)

Added:
  • UI:
    • Update download progress indication and cancelling possibility in SpectX Update dialog.
    • Data Store editor: glob patterns can be used in data store Read ACL definitions, in addition to prefixed uri paths.
    • Resource tree filter. Defaults to filename filter, c:<string> searches for files containing ‘string’, and t:<hour/day/week> searches for files modified last hour/day/week.
    • Result set _raw_text column contents exportable to file from ‘Save’ and right click context menus
    • Query filter creation from resultset selection supports time ranges for timestamp columns
  • Support for reading data from ZIP archives. See more at Working With Archives.
  • UserAdmin role.
Changed:
  • Engine:
    • Increased processing speed of Bzip2 files on Intel-based 64 bit OSX and Linux platforms by 20%.
  • UI:
    • Improved editor auto-completion in nested function calls
    • Data Store Editor: ‘.’ symbol is allowed in store name
Fixed:
  • UI:
    • Map feature properties popup displays correct values for tuple type columns
    • Export of chart with column names containing non-ASCII characters

v1.4.31 (2019-02-22)

Changed:
  • SpectX configuration keys engine.fs_access and engine.fs_unmanaged_access are deprecated. Instead, new configuration keys engine.da.protocol.<protocol> can be used to define usage permissions for each data access protocol separately.

v1.4.30 (2019-02-11)

Added:
  • DataAccess/HDFS: High availability clusters accessibility support

v1.4.29 (2019-02-08)

Added:
  • Functions: INDEXOF, ROT13, ESCAPE_OCTAL
Changed:
  • OpenJDK 8 HotSpot Java Runtime Environment is now supported
  • DataAccess/HDFS: improved accessing secure HDFS clusters with multiple Kerberos authentication realms configured
Fixed:
  • On Windows, Data Browser started by pressing Browse button in data store configuration editor window for previously stored configuration, fails to connect to the data store.

v1.4.28 (2019-01-10)

This release contains bug fixes, a few new features and some improvements.

Added:
Changed:
  • Reading data from relational databases: refactored input stream commands
  • UI:
    • Data Store editor: the browse-button uses real-time values in the editor form to connect to the data store. Browsing is restricted to the currently editable store only.
    • DataBrowser: keyboard navigation improved. Arrow, Pg Down/Up, Enter and Backspace keys can be used for navigation in all views.
    • “Admin - Users and Groups” menu: added descriptive text to the User/Group/ExtGroupMapping deletion dialogs.
    • “System status” menu: remastered Effective Settings view.
    • Script and Pattern editors: removed ctrl - shift - arrow tab navigation, text of displayed error messages made selectable.
    • The full name of a logged in user (if set) is used instead of a username in the rightmost menu at the top bar
    • Background version checker: connection exception logging with debug log level (previously: error log level).
  • Engine:
    • Improved processing performance of concatenated GZ/ZZ/PiGZ/PiZZ files.
    • Increased processing speed of GZ/ZZ/PiGZ/PiZZ files on Intel-based 64 bit OSX and Linux platforms by 20%.
Fixed:
  • Listing queries for cloud (Amazon/Azure/Google cloud) targets with the ‘includeContainers’:true argument.
  • Removed unnecessary creation of the folder “users” under the path specified by the config parameter sx.user_data.dir.
  • Failure to attempt to create own log directory on the very first run fixed in both SpectX and SourceAgent startup Windows scripts.