🚀 Features
Section titled “🚀 Features”Implement a zeek-tsv format
Section titled “Implement a zeek-tsv format”May 11, 2023 · @Dakostu · #3114
The zeek-tsv format parses and prints Zeek’s native tab-separated value (TSV)
representation of logs.
Implement a distributed pipeline executor
Section titled “Implement a distributed pipeline executor”May 6, 2023 · @dominiklohmann · #3119
Pipelines may now span across multiple processes. This will enable upcoming
operators that do not just run locally in the vast exec process, but rather
connect to a VAST node and partially run in that node. The new operator
modifiers remote and local allow expert users to control where parts of
their pipeline run explicitly, e.g., to offload compute to a more powerful node.
Potentially unsafe use of these modifiers requires setting
vast.allow-unsafe-pipelines to true in the configuration file.
Introduce a version source operator
Section titled “Introduce a version source operator”May 5, 2023 · @dominiklohmann · #3123
The vast exec command now supports implicit sinks for pipelines that end in
events or bytes: write json --pretty and save file -, respectively.
The --pretty option for the JSON printer enables multi-line output.
The new version source operator yields a single event containing VAST’s
version and a list of enabled plugins.
Implement the measure operator
Section titled “Implement the measure operator”May 3, 2023 · @dominiklohmann · #3093
The inspect operator replaces the events or bytes it receives with incremental
metrics describing the input.
Add new CEF parser plugin
Section titled “Add new CEF parser plugin”Apr 29, 2023 · @jachris · #3110
The cef parser allows for using the CEF format with the new pipelines.
Add Feather and Parquet parsers and printers
Section titled “Add Feather and Parquet parsers and printers”Apr 28, 2023 · @dominiklohmann · #3103
The feather and parquet formats allow for reading and writing events from
and to the Apache Feather V2 and Apache Parquet files, respectively.
Implement xsv parser & printer
Section titled “Implement xsv parser & printer”Apr 27, 2023 · @Dakostu · #3104
The xsv format enables the user to parse and print character-separated
values, with the additional csv, tsv and ssv formats as sane defaults.
Add directory saver
Section titled “Add directory saver”Apr 25, 2023 · @Dakostu · #3098
The new directory sink creates a directory with a file for each schema in
the specified format.
PRs 3085-3088-3097
Section titled “PRs 3085-3088-3097”Apr 20, 2023 · @jachris · #3085
The new file connector enables the user to process file input/output as data
in a pipeline. This includes regular files, UDS files as well as
stdin/stdout.
Upgrade exporter to use new pipelines
Section titled “Upgrade exporter to use new pipelines”Apr 19, 2023 · @jachris · #3076
The vast export command now accepts the new pipelines as input. Furthermore,
vast export <expr> is now deprecated in favor of vast export 'where <expr>'.
Expose the lower-level load, parse, print, and save operators
Section titled “Expose the lower-level load, parse, print, and save operators”Apr 17, 2023 · @dominiklohmann · #3079
The new from <connector> [read <format>], read <format> [from <connector>],
write <format> [to <connector>], and to <connector> [write <format>]
operators bring together a connector and a format to prduce and consume events,
respectively. Their lower-level building blocks load <connector>, parse <format>, print <format>, and save <connector> enable expert users to
operate on raw byte streams directly.
Upgrade partition transformer to new pipelines
Section titled “Upgrade partition transformer to new pipelines”Apr 13, 2023 · @jachris · #3064
User-defined operator aliases make pipelines easier to use by enabling users to
encapsulate a pipelinea into a new operator. To define a user-defined operator
alias, add an entry to the vast.operators section of your configuration.
Compaction now makes use of the new pipeline operators, and allows pipelines to
be defined inline instead in addition to the now deprecated vast.pipelines
configuration section.
Introduce the count_distinct aggregation function
Section titled “Introduce the count_distinct aggregation function”Apr 12, 2023 · @dominiklohmann · #3068
The count_distinct aggregation function returns the number of distinct,
non-null values.
Add unique operator
Section titled “Add unique operator”Apr 12, 2023 · @jachris · #3051
The newly-added unique operator removes adjacent duplicates.
Add tail operator
Section titled “Add tail operator”Apr 2, 2023 · @Dakostu · #3050
The new tail pipeline operator limits all latest events to a specified
number. The operator takes the limit as an optional argument, with the default
value being 10.
PRs 3036-3039-3089
Section titled “PRs 3036-3039-3089”Mar 29, 2023 · @dominiklohmann · #3036
The put operator is the new companion to the existing extend and replace
operators. It specifies the output fields exactly, referring either to input
fields with an extractor, metadata with a selector, or a fixed value.
The extend and replace operators now support assigning extractors and
selectors in addition to just fixed values.
🔧 Changes
Section titled “🔧 Changes”Upgrade exporter to use new pipelines
Section titled “Upgrade exporter to use new pipelines”Apr 19, 2023 · @jachris · #3076
The exporter.* metrics no longer exist, and will return in a future release as
a more generic instrumentation mechanism for all pipelines.
Add support for user-defined operator aliases
Section titled “Add support for user-defined operator aliases”Apr 12, 2023 · @jachris · #3067
The vast.operators section in the configuration file supersedes the now
deprecated vast.pipelines section and more generally enables user-defined
operators. Defined operators now must use the new, textual format introduced
with VAST v3.0, and are available for use in all places where pipelines
are supported.
Restart the systemd service on failure
Section titled “Restart the systemd service on failure”The bundled systemd service is now configured to restart VAST in case of a failure.
Remove configuration-defined import/export pipelines
Section titled “Remove configuration-defined import/export pipelines”Apr 3, 2023 · @dominiklohmann · #3052
As already announced with the VAST v3.0 release, the vast.pipeline-triggers
option now no longer functions. The feature will be replaced with node
ingress/egress pipelines that fit better into a multi-node model than the
previous feature that was built under the assumption of a client/server model
with a single server.
Update query endpoint to use new pipeline executor
Section titled “Update query endpoint to use new pipeline executor”Mar 31, 2023 · @jachris · #3015
The /query REST endpoint no longer accepts an expression at the start of the
query. Instead, use where <expr> | ....
🐞 Bug Fixes
Section titled “🐞 Bug Fixes”Fix remaining partitions counter in the rebuilder
Section titled “Fix remaining partitions counter in the rebuilder”May 12, 2023 · @dominiklohmann · #3147
The rebuilder.partitions.remaining metric sometimes reported wrong values when
partitions for at least one schema did not need to be rebuilt. We aligned the
metrics with the actual functionality.
Set minimum timestamp of partitions properly
Section titled “Set minimum timestamp of partitions properly”May 11, 2023 · @dominiklohmann · #3141
Some pipelines in compaction caused transformed partitions to be treated as if they were older than they were supposed to be, causing them to be picked up again for deletion too early. This bug no longer exists, and compacted partitions are now considered at most as old as the oldest event before compaction.
Align endpoints between regular and slim Docker images
Section titled “Align endpoints between regular and slim Docker images”May 10, 2023 · @dominiklohmann · #3137
The tenzir/vast image now listens on 0.0.0.0:5158 instead of
127.0.0.1:5158 by default, which aligns the behavior with the
tenzir/vast-slim image.
Bump vast-plugins to a95e420
Section titled “Bump vast-plugins to a95e420”The matcher plugin no longer causes deadlocks through detached matcher clients.
Mark some CAF types as nodiscard
Section titled “Mark some CAF types as nodiscard”Apr 19, 2023 · @dominiklohmann · #3086
Tokens created with vast web generate-token now persist correctly, and work
across restarts of VAST.
Introduce the count_distinct aggregation function
Section titled “Introduce the count_distinct aggregation function”Apr 12, 2023 · @dominiklohmann · #3068
The distinct function silently performed a different operation on lists,
returning the distinct non-null elements in the list rather than operating on
the list itself. This special-casing no longer exists, and instead the function
now operates on the lists itself. This feature will return in the future as
unnesting on the extractor level via distinct(field[]), but for now it has to
go to make the distinct aggregation function work consistently.
Fix subnet queries for some subnets
Section titled “Fix subnet queries for some subnets”Apr 6, 2023 · @dominiklohmann · #3060
VAST incorrectly handled subnets using IPv6 addresses for which an equivalent
IPv4 address existed. This is now done correctly. For example, the query where :ip !in ::ffff:0:0/96 now returns all events containing an IP address that
cannot be represented as an IPv4 address. As an additional safeguard, the VAST
language no longer allows for constructing subnets for IPv4 addresses with
lengths greater than 32.