Download the release on GitHub.
Features
Section titled “Features”Introduce a version source operator
Section titled “Introduce a version source operator”The vast exec command now supports implicit sinks for pipelines that end in
events or bytes: write json --pretty and save file -, respectively.
The --pretty option for the JSON printer enables multi-line output.
The new version source operator yields a single event containing VAST’s
version and a list of enabled plugins.
By @dominiklohmann in #3123.
Implement the measure operator
Section titled “Implement the measure operator”The inspect operator replaces the events or bytes it receives with incremental
metrics describing the input.
By @dominiklohmann in #3093.
PRs 3036-3039-3089
Section titled “PRs 3036-3039-3089”The put operator is the new companion to the existing extend and replace
operators. It specifies the output fields exactly, referring either to input
fields with an extractor, metadata with a selector, or a fixed value.
The extend and replace operators now support assigning extractors and
selectors in addition to just fixed values.
By @dominiklohmann in #3036.
Add directory saver
Section titled “Add directory saver”The new directory sink creates a directory with a file for each schema in
the specified format.
Introduce the count_distinct aggregation function
Section titled “Introduce the count_distinct aggregation function”The count_distinct aggregation function returns the number of distinct,
non-null values.
By @dominiklohmann in #3068.
Expose the lower-level load, parse, print, and save operators
Section titled “Expose the lower-level load, parse, print, and save operators”The new from <connector> [read <format>], read <format> [from <connector>],
write <format> [to <connector>], and to <connector> [write <format>]
operators bring together a connector and a format to prduce and consume events,
respectively. Their lower-level building blocks load <connector>, parse <format>, print <format>, and save <connector> enable expert users to
operate on raw byte streams directly.
By @dominiklohmann in #3079.
Add unique operator
Section titled “Add unique operator”The newly-added unique operator removes adjacent duplicates.
Add Feather and Parquet parsers and printers
Section titled “Add Feather and Parquet parsers and printers”The feather and parquet formats allow for reading and writing events from
and to the Apache Feather V2 and Apache Parquet files, respectively.
By @dominiklohmann in #3103.
Implement xsv parser & printer
Section titled “Implement xsv parser & printer”The xsv format enables the user to parse and print character-separated
values, with the additional csv, tsv and ssv formats as sane defaults.
Implement a distributed pipeline executor
Section titled “Implement a distributed pipeline executor”Pipelines may now span across multiple processes. This will enable upcoming
operators that do not just run locally in the vast exec process, but rather
connect to a VAST node and partially run in that node. The new operator
modifiers remote and local allow expert users to control where parts of
their pipeline run explicitly, e.g., to offload compute to a more powerful node.
Potentially unsafe use of these modifiers requires setting
vast.allow-unsafe-pipelines to true in the configuration file.
By @dominiklohmann in #3119.
Add new CEF parser plugin
Section titled “Add new CEF parser plugin”The cef parser allows for using the CEF format with the new pipelines.
Upgrade exporter to use new pipelines
Section titled “Upgrade exporter to use new pipelines”The vast export command now accepts the new pipelines as input. Furthermore,
vast export <expr> is now deprecated in favor of vast export 'where <expr>'.
Implement a zeek-tsv format
Section titled “Implement a zeek-tsv format”The zeek-tsv format parses and prints Zeek’s native tab-separated value (TSV)
representation of logs.
Upgrade partition transformer to new pipelines
Section titled “Upgrade partition transformer to new pipelines”User-defined operator aliases make pipelines easier to use by enabling users to
encapsulate a pipelinea into a new operator. To define a user-defined operator
alias, add an entry to the vast.operators section of your configuration.
Compaction now makes use of the new pipeline operators, and allows pipelines to
be defined inline instead in addition to the now deprecated vast.pipelines
configuration section.
Add tail operator
Section titled “Add tail operator”The new tail pipeline operator limits all latest events to a specified
number. The operator takes the limit as an optional argument, with the default
value being 10.
PRs 3085-3088-3097
Section titled “PRs 3085-3088-3097”The new file connector enables the user to process file input/output as data
in a pipeline. This includes regular files, UDS files as well as
stdin/stdout.
Changes
Section titled “Changes”Restart the systemd service on failure
Section titled “Restart the systemd service on failure”The bundled systemd service is now configured to restart VAST in case of a failure.
Add support for user-defined operator aliases
Section titled “Add support for user-defined operator aliases”The vast.operators section in the configuration file supersedes the now
deprecated vast.pipelines section and more generally enables user-defined
operators. Defined operators now must use the new, textual format introduced
with VAST v3.0, and are available for use in all places where pipelines
are supported.
Upgrade exporter to use new pipelines
Section titled “Upgrade exporter to use new pipelines”The exporter.* metrics no longer exist, and will return in a future release as
a more generic instrumentation mechanism for all pipelines.
Update query endpoint to use new pipeline executor
Section titled “Update query endpoint to use new pipeline executor”The /query REST endpoint no longer accepts an expression at the start of the
query. Instead, use where <expr> | ....
Remove configuration-defined import/export pipelines
Section titled “Remove configuration-defined import/export pipelines”As already announced with the VAST v3.0 release, the vast.pipeline-triggers
option now no longer functions. The feature will be replaced with node
ingress/egress pipelines that fit better into a multi-node model than the
previous feature that was built under the assumption of a client/server model
with a single server.
By @dominiklohmann in #3052.
Bug Fixes
Section titled “Bug Fixes”Introduce the count_distinct aggregation function
Section titled “Introduce the count_distinct aggregation function”The distinct function silently performed a different operation on lists,
returning the distinct non-null elements in the list rather than operating on
the list itself. This special-casing no longer exists, and instead the function
now operates on the lists itself. This feature will return in the future as
unnesting on the extractor level via distinct(field[]), but for now it has to
go to make the distinct aggregation function work consistently.
By @dominiklohmann in #3068.
Mark some CAF types as nodiscard
Section titled “Mark some CAF types as nodiscard”Tokens created with vast web generate-token now persist correctly, and work
across restarts of VAST.
By @dominiklohmann in #3086.
Fix subnet queries for some subnets
Section titled “Fix subnet queries for some subnets”VAST incorrectly handled subnets using IPv6 addresses for which an equivalent
IPv4 address existed. This is now done correctly. For example, the query where :ip !in ::ffff:0:0/96 now returns all events containing an IP address that
cannot be represented as an IPv4 address. As an additional safeguard, the VAST
language no longer allows for constructing subnets for IPv4 addresses with
lengths greater than 32.
By @dominiklohmann in #3060.
Set minimum timestamp of partitions properly
Section titled “Set minimum timestamp of partitions properly”Some pipelines in compaction caused transformed partitions to be treated as if they were older than they were supposed to be, causing them to be picked up again for deletion too early. This bug no longer exists, and compacted partitions are now considered at most as old as the oldest event before compaction.
By @dominiklohmann in #3141.
Align endpoints between regular and slim Docker images
Section titled “Align endpoints between regular and slim Docker images”The tenzir/vast image now listens on 0.0.0.0:5158 instead of
127.0.0.1:5158 by default, which aligns the behavior with the
tenzir/vast-slim image.
By @dominiklohmann in #3137.
Fix remaining partitions counter in the rebuilder
Section titled “Fix remaining partitions counter in the rebuilder”The rebuilder.partitions.remaining metric sometimes reported wrong values when
partitions for at least one schema did not need to be rebuilt. We aligned the
metrics with the actual functionality.
By @dominiklohmann in #3147.
Bump vast-plugins to a95e420
Section titled “Bump vast-plugins to a95e420”The matcher plugin no longer causes deadlocks through detached matcher clients.