🚀 Features
Section titled “🚀 Features”Implement a generic aggregation transform step
Section titled “Implement a generic aggregation transform step”Feb 15, 2022 · @dominiklohmann · #2076
The new built-in rename transform step allows for renaming event types
during a transformation. This is useful when you want to ensure that a
repeatedly triggered transformation does not affect already transformed
events.
The new aggregate transform plugin allows for flexibly grouping and
aggregating events. We recommend using it alongside the compaction
plugin, e.g., for rolling
up events into a more space-efficient representation after a certain amount of
time.
Add new query language plugin
Section titled “Add new query language plugin”VAST has a new query language plugin type that allows for adding additional
query language frontends. The plugin performs one function: compile user input
into a VAST expression. The new sigma plugin demonstrates usage of this plugin
type.
PRs 2064-2082
Section titled “PRs 2064-2082”The built-in select and project transform steps now correctly handle
dropping all rows and columns respectively, effectively deleting the input
data.
🔧 Changes
Section titled “🔧 Changes”Correctly handle partition transforms without output
Section titled “Correctly handle partition transforms without output”We fixed an issue where partition transforms that erase complete partitions trigger an internal assertion failure.
Log actor names together with the unique actor ID
Section titled “Log actor names together with the unique actor ID”Actor names in log messages now have an -ID suffix to make it easier to tell
multiple instances of the same actor apart, e.g., exporter-42.
Deprecate the msgpack table slice
Section titled “Deprecate the msgpack table slice”The msgpack encoding option is now deprecated. VAST issues a warning on
startup and automatically uses the arrow encoding instead. A future version of
VAST will remove this option entirely.
The experimental aging feature is now deprecated. The compaction plugin offers a superset of the aging functionality.
Add new query language plugin
Section titled “Add new query language plugin”VAST no longer attempts to intepret query expressions as Sigma rules
automatically. Instead, this functionality moved to a dedicated sigma query
language plugin that must explicitly be enabled at build time.
🐞 Bug Fixes
Section titled “🐞 Bug Fixes”Send initial db state to new partition creation listeners
Section titled “Send initial db state to new partition creation listeners”We fixed a bug that potentially resulted in the wrong subset of partitions to be considered during query evaluation.
Adjust index statistics for partition transforms
Section titled “Adjust index statistics for partition transforms”VAST now correctly adjusts the index statistics when applying partition transforms.
Fix hanging queries
Section titled “Fix hanging queries”VAST does not lose query capacity when backlogged queries are cancelled any more.
Improve name lookup in meta index to reduce FPs
Section titled “Improve name lookup in meta index to reduce FPs”Feb 16, 2022 · @dominiklohmann · #2086
A performance bug in the first stage of query evaluation caused VAST to return
too many candidate partitions when querying for a field suffix. For example, a
query for the ts field commonly used in Zeek logs also included partitions for
netflow.pkts from suricata.netflow events. This bug no longer exists,
resulting in a considerable speedup of affected queries.