Download the release on GitHub.
Features
Section titled “Features”PRs 1330-1376
Section titled “PRs 1330-1376”The meta index now stores partition synopses in separate files. This will
decrease restart times for systems with large databases, slow disks and
aggressive readahead settings. A new config setting vast.meta-index-dir
allows storing the meta index information in a separate directory.
By @dominiklohmann in #1330.
Enable real-time metrics reporting
Section titled “Enable real-time metrics reporting”The new options vast.metrics.file-sink.real-time and
vast.metrics.uds-sink.real-time enable real-time metrics reporting for the
file sink and UDS sink respectively.
By @dominiklohmann in #1368.
PRs 1343-1356-ngrodzitski
Section titled “PRs 1343-1356-ngrodzitski”The JSON import now always relies upon simdjson. The
previously experimental --simdjson option to the vast import json|suricata|zeek-json commands no longer exist as the feature is considered
stable.
By @ngrodzitski in #1343.
PRs 1223-1362
Section titled “PRs 1223-1362”VAST rotates server logs by default. The new config options
vast.disable-log-rotation and vast.log-rotation-threshold can be used to
control this behaviour.
By @dominiklohmann in #1223.
Enable type extractors to support user defined types
Section titled “Enable type extractors to support user defined types”The type extractor in the expression language now works with user defined types.
For example the type port is defined as type port = count in the base
schema. This type can now be queried with an expression like :port == 80.
Add native Sigma support
Section titled “Add native Sigma support”Sigma rules are now a valid format to
represent query expression. VAST parses the detection attribute of a rule and
translates it into a native query expression. To run a query using a Sigma rule,
pass it on standard input, e.g., vast export json < rule.yaml.
Changes
Section titled “Changes”Render help and documentation on stdout
Section titled “Render help and documentation on stdout”The output of vast help and vast documentation now goes to stdout instead
of to stderr. Erroneous invocations of vast also print the helptext, but in
this case the output still goes to stderr to avoid interference with downstream
tooling.
PRs 1343-1356-ngrodzitski
Section titled “PRs 1343-1356-ngrodzitski”The infer command has an improved heuristic for the number types int,
count, and real.
By @ngrodzitski in #1343.
Enable real-time metrics reporting
Section titled “Enable real-time metrics reporting”All options in vast.metrics.* had underscores in their names replaced with
dashes to align with other options. For example, vast.metrics.file_sink is now
vast.metrics.file-sink. The old options no longer work.
By @dominiklohmann in #1368.
PRs 1223-1328-1334-1390-a4z
Section titled “PRs 1223-1328-1334-1390-a4z”VAST switched to spdlog >= 1.5.0 for
logging. For users, this means: The vast.console-format and vast.file-format
now must be specified using the spdlog pattern syntax as described
here.
All settings under caf.logger.* are now ignored by VAST, and only the vast.*
counterparts are used for logger configuration.
By @dominiklohmann in #1223.
Move options from format to the import subcommand
Section titled “Move options from format to the import subcommand”The options listen, read, schema, schema-file, type, and uds can
from now on be supplied to the import command directly. Similarly, the options
write and uds can be supplied to the export command. All options can still
be used after the format subcommand, but that usage is deprecated.
Expand subnet value predicates
Section titled “Expand subnet value predicates”The query normalizer interprets value predicates of type subnet more broadly:
given a subnet S, the parser expands this to the expression :subnet == S || :addr in S. This change makes it easier to search for IP addresses belonging to
a specific subnet.
Read user-supplied schema files from config dirs
Section titled “Read user-supplied schema files from config dirs”User-supplied schema files are now picked up from <SYSCONFDIR>/vast/schema and
<XDG_CONFIG_HOME>/vast/schema instead of <XDG_DATA_HOME>/vast/schema.
By @dominiklohmann in #1372.
Remove long-deprecated code
Section titled “Remove long-deprecated code”The previously deprecated options vast.spawn.importer.ids and
vast.schema-paths no longer work. Furthermore, queries spread over multiple
arguments are now disallowed instead of triggering a deprecation warning.
By @dominiklohmann in #1374.
Require fmt to be installed separately from spdlog
Section titled “Require fmt to be installed separately from spdlog”VAST now requires fmt >= 5.2.1 to be installed.
By @dominiklohmann in #1330.
Introduce and make use of the timestamp type
Section titled “Introduce and make use of the timestamp type”The special meaning of the #timestamp attribute has been removed from the
schema language. Timestamps can from now on be marked as such by using the
timestamp type instead. Queries of the form #timestamp <op> value remain
operational but are deprecated in favor of :timestamp. Note that this change
also affects :time queries, which aren’t supersets of #timestamp queries any
longer.
Make it easier to reference user defined types in the schema language
Section titled “Make it easier to reference user defined types in the schema language”Schema parsing now uses a 2-pass loading phase so that type aliases can reference other types that are later defined in the same directory. Additionally, type definitions from already parsed schema dirs can be referenced from schema types that are parsed later. Types can also be redefined in later directories, but a type can not be defined twice in the same directory.
Bug Fixes
Section titled “Bug Fixes”Revert “Fix potential race condition between evaluator and partition”
Section titled “Revert “Fix potential race condition between evaluator and partition””An ordering issue introduced in #1295 that could lead to a segfault with long-running queries was reverted.
Let the JSON reader recover from unexpected inputs
Section titled “Let the JSON reader recover from unexpected inputs”A bug in the new simdjson based JSON reader introduced in
#1356 could trigger an assertion in
the vast import process if an input field could not be converted to the field
type in the target layout. This is no longer the case.