A record stream is the generic form of structured data used by SpectX for query processing. The structure (type and constraints) of a record is determined by exported fields in the pattern used to extract and transform parse the raw input data. In the case of querying data from structured formats (AVRO, ROSBAG) the types of fields are converted to SpectX data types. Hence a SpectX record is quite the same as a record in a relational database.
The following commands create a record stream:
- LIST performs listing of input resources, producing record stream of file metadata
- PARSE performs extract and transform of exported fields of the specified pattern
- GREP searches a given string in given input files
- JDBC_TABLE and JDBC_QUERY fetch data from relational databases
- ES fetches data from Elasticsearch storage
- ROSBAG fetches data from Robot Operating System bag files
- AVRO fetches data from Apache AVRO files
- VALUE converts input string arguments to desired types, producing a set of one or more records
- DUAL produces records with various type of fields using built-in data generator
An existing record stream can be referred and manipulated by query statements.