Download the release on GitHub.
Features
Section titled “Features”PRs 1196-1233
Section titled “PRs 1196-1233”The new dump command prints configuration and schema-related information. The
implementation allows for printing all registered concepts and models, via vast dump concepts and vast dump models. The flag to --yaml to dump switches
from JSON to YAML output, such that it confirms to the taxonomy configuration
syntax.
By @dominiklohmann in #1196.
Add support for type-level synopses and a string synopsis
Section titled “Add support for type-level synopses and a string synopsis”Low-selectivity queries of string (in)equality queries now run up to 30x faster, thanks to more intelligent selection of relevant index partitions.
Add support for USDT tracepoints in VAST
Section titled “Add support for USDT tracepoints in VAST”On Linux, VAST now contains a set of built-in USDT tracepoints that can be used
by tools like perf or bpftrace when debugging. Initially, we provide the two
tracepoints chunk_make and chunk_destroy, which trigger every time a
vast::chunk is created or destroyed.
PRs 1135-1150
Section titled “PRs 1135-1150”VAST now ships with its own taxonomy and basic concept definitions for Suricata, Zeek, and Sysmon.
Show file size information in lsvast
Section titled “Show file size information in lsvast”The new option --print-bytesizes of lsvast prints information about the size
of certain fields of the flatbuffers inside a VAST database directory.
Introduce the #field meta extractor
Section titled “Introduce the #field meta extractor”The expression language gained support for the #field meta extractor. It is
the complement for #type and uses suffix matching for field names at the
layout level.
PRs 1172-1200-1216
Section titled “PRs 1172-1200-1216”The storage required for index IP addresses has been optimized. This should result in significantly reduced memory usage over time, as well as faster restart times and reduced disk space requirements.
Allow for enabling client file logging
Section titled “Allow for enabling client file logging”The new option vast.client-log-file enables client-side logging. By default,
VAST only writes log files for the server process.
By @dominiklohmann in #1132.
Show meta index size in vast status
Section titled “Show meta index size in vast status”A new key ‘meta-index-bytes’ appears in the status output generated by vast status --detailed.
PRs 1185-1228
Section titled “PRs 1185-1228”The query language now supports models. Models combine a list of concepts into a semantic unit that can be fulfiled by an event. If the type of an event contains a field for every concept in a model. Turn to the documentation for more information.
Changes
Section titled “Changes”Make Zeek writer work with all data types
Section titled “Make Zeek writer work with all data types”The zeek export format now strips off the prefix zeek. to ensure full
compatibility with regular Zeek output. For all non-Zeek types, the prefix
remains intact.
Process schema directories recursively
Section titled “Process schema directories recursively”VAST now processes the schema directory recursively, as opposed to stopping at nested directories.
Make metrics opt-in
Section titled “Make metrics opt-in”VAST does not produce metrics by default any more. The option
--disable-metrics has been renamed to --enable-metrics accordingly.
PRs 1176-1180-1186-1237-satta
Section titled “PRs 1176-1180-1186-1237-satta”The Suricata schemas received an overhaul: there now exist vlan and in_iface
fields in all types. In addition, VAST ships with new types for ikev2, nfs,
snmp, tftp, rdp, sip and dcerpc. The tls type gets support for the
additional sni and session_resumed fields.
Move schema definitions into subdirectory
Section titled “Move schema definitions into subdirectory”Installed schema definitions now reside in <datadir>/vast/schema/types,
taxonomy definitions in <datadir>/vast/schema/taxonomy, and concept
definitions in <datadir/vast/schema/concepts, as opposed to them all being in
the schema directory directly. When overriding an existing installation, you
may have to delete the old schema definitions by hand.
By @dominiklohmann in #1194.
Set fallback port for underspecified endpoints
Section titled “Set fallback port for underspecified endpoints”VAST now listens on port 42000 instead of letting the operating system choose
the port if the option vast.endpoint specifies an endpoint without a port. To
restore the old behavior, set the port to 0 explicitly.
By @dominiklohmann in #1170.
Fulfill Deployment Requirements
Section titled “Fulfill Deployment Requirements”The build configuration of VAST received a major overhaul. Inclusion of libvast
in other procects via add_subdirectory(path/to/vast) is now easily possible.
The names of all build options were aligned, and the new build summary shows all
available options.
By @dominiklohmann in #1175.
Replace PID file if process does not exist
Section titled “Replace PID file if process does not exist”VAST no longer requires you to manually remove a stale PID file from a no-longer
running vast process. Instead, VAST prints a warning and overwrites the old
PID file.
PRs 1143-1157-1160-1165
Section titled “PRs 1143-1157-1160-1165”The on-disk format for table slices now supports versioning of table slice encodings. This breaking change makes it so that adding further encodings or adding new versions of existing encodings is possible without breaking again in the future.
By @dominiklohmann in #1143.
Generalize splunk-to-vast
Section titled “Generalize splunk-to-vast”The splunk-to-vast script has a new name: taxonomize. The script now also
generates taxonomy declarations for Azure Sentinel.
Remove port type
Section titled “Remove port type”The port type is no longer a first-class type. The new way to represent
transport-layer ports relies on count instead. In the schema, VAST ships with
a new alias type port = count to keep existing schema definitions in tact.
However, this is a breaking change because the on-disk format and Arrow data
representation changed. Queries with :port type extractors no longer work.
Similarly, the syntax 53/udp no longer exists; use count syntax 53
instead. Since most port occurrences do not carry a known transport-layer
type, and the type information exists typically in a separate field, removing
port as native type streamlines the data model.
Remove Version FlatBuffers table
Section titled “Remove Version FlatBuffers table”Archive segments no longer include an additional, unnecessary version identifier. We took the opportunity to clean this up bundled with the other recent breaking changes.
By @dominiklohmann in #1168.
Remove CAF-encoded table slices
Section titled “Remove CAF-encoded table slices”CAF-encoded table slices no longer exist. As such, the option
vast.import.batch-encoding now only supports arrow and msgpack as
arguments.
By @dominiklohmann in #1142.
Increase the default segment size to 1 GiB
Section titled “Increase the default segment size to 1 GiB”The default segment size in the archive is now 1 GiB. This reduces fragmentation of the archive meta data and speeds up VAST startup time.
Bug Fixes
Section titled “Bug Fixes”Reply to status requests in sinks
Section titled “Reply to status requests in sinks”The output of vast status --detailed now contains informations about runnings
sinks, e.g., vast export <format> <query> processes.
By @dominiklohmann in #1155.
Send correct message to index when dropping further results
Section titled “Send correct message to index when dropping further results”The index now correctly drops further results when queries finish early, thus improving the performance of queries for a limited number of events.
Error when specified config file does not exist
Section titled “Error when specified config file does not exist”VAST no longer starts if the specified config file does not exist.
By @dominiklohmann in #1147.
Introduce the #field meta extractor
Section titled “Introduce the #field meta extractor”The summary log message of vast export now contains the correct number of
candidate events.
Fix index worker depletion
Section titled “Fix index worker depletion”The index no longer causes exporters to deadlock when the meta index produces false positives.
Fix YAML syntax errors in example config
Section titled “Fix YAML syntax errors in example config”The vast.yaml.example contained syntax errors. The example config file now
works again.
Fix loading and dumping of composed concepts
Section titled “Fix loading and dumping of composed concepts”Concepts that reference other concepts are now loaded correctly from their definition.
Detect and handle breaking changes in schemas
Section titled “Detect and handle breaking changes in schemas”The type registry now detects and handles breaking changes in schemas, e.g., when a field type changes or a field is dropped from record.
By @dominiklohmann in #1195.
Fix a hang when trying to process an invalid query
Section titled “Fix a hang when trying to process an invalid query”VAST no longer blocks when an invalid query operation is issued.
Fix a bug that causes sources to stall
Section titled “Fix a bug that causes sources to stall”vast import no longer stalls when it doesn’t receive any data for more than 10
seconds.
By @dominiklohmann in #1136.
Don’t collect status from sources and sinks
Section titled “Don’t collect status from sources and sinks”The vast status command does not collect status information from sources and
sinks any longer. They were often too busy to respond, leading to a long delay
before the command completed.
Switch index behavior when running out of workers
Section titled “Switch index behavior when running out of workers”The index no longer crashes when too many parallel queries are running.
By @dominiklohmann in #1210.