🚀 Features
Section titled “🚀 Features”Support pattern case insensitivity in Sigma plugin
Section titled “Support pattern case insensitivity in Sigma plugin”Mar 3, 2023 · @Dakostu · #2974
The sigma plugin now treats Sigma strings as case-insensitive patterns during
the transpilation process.
Add extractor predicates
Section titled “Add extractor predicates”Mar 2, 2023 · @jachris · #2984
Extractors such as x and :T can now expand to the predicates x != null
and :T != null, respectively.
Add a new /openapi.json endpoint to the web plugin
Section titled “Add a new /openapi.json endpoint to the web plugin”The experimental web plugin now serves its own API specification at the new ‘/openapi.json’ endpoint.
Support case insensitivity in patterns
Section titled “Support case insensitivity in patterns”Feb 17, 2023 · @Dakostu · #2951
Patterns now support case insensitivity by adding i to the pattern string,
e.g. /^\w{3}$/i.
Add CORS preflight request handling
Section titled “Add CORS preflight request handling”The experimental web frontend now correctly responds to CORS preflight requests.
To configure CORS behavior, the new vast.web.cors-allowed-origin config option
can be used.
Implement head and taste operators
Section titled “Implement head and taste operators”Jan 30, 2023 · @dominiklohmann · #2891
The new head and taste operators limit results to the specified number of
events. The head operator applies this limit for all events, and the taste
operator applies it per schema. Both operators take the limit as an optional
argument, with the default value being 10.
PRs 2877-2904-2907
Section titled “PRs 2877-2904-2907”Jan 27, 2023 · @Dakostu · #2877
The export and import commands now support an optional pipeline string
that allows for chaining pipeline operators together and executing such a
pipeline on outgoing and incoming data. This feature is experimental and the
syntax is subject to change without notice. New operators are only available in
the new pipeline syntax, and the old YAML syntax is deprecated.
Implement a retry mechanism for VAST clients failing to connect to the server
Section titled “Implement a retry mechanism for VAST clients failing to connect to the server”Jan 23, 2023 · @patszt · #2835
We changed VAST client processes to attempt connecting to a VAST server multiple
times until the configured connection timeout (vast.connection-timeout,
defaults to 5 minutes) runs out. A fixed delay between connection attempts
(vast.connection-retry-delay, defaults to 3 seconds) ensures that clients to
not stress the server too much. Set the connection timeout to zero to let VAST
client attempt connecting indefinitely, and the delay to zero to disable the
retry mechanism.
Add options to omit empty values when exporting as JSON
Section titled “Add options to omit empty values when exporting as JSON”Jan 17, 2023 · @dominiklohmann · #2856
The JSON export format gained the options --omit-empty-records,
--omit-empty-lists, and --omit-empty-maps, which cause empty records, lists,
and maps not to be rendered respectively. The options may be combined together
with the existing --omit-nulls option. Use --omit-empty to set all four
flags at once.
Install Python bindings along with VAST
Section titled “Install Python bindings along with VAST”Jan 13, 2023 · @dominiklohmann · #2636
VAST installations and packages now include Python bindings in a site-package
under <install-prefix>/lib/python*/site-packages/vast.
Add ‘pipeline’ parameter and schematized format to export endpoint
Section titled “Add ‘pipeline’ parameter and schematized format to export endpoint”The /export family of endpoints now accepts an optional pipeline
parameter to specify an ad-hoc pipeline that should be applied to
the exported data.
Make it easy to create docker images with Nix
Section titled “Make it easy to create docker images with Nix”We now offer a tenzir/vast-slim image as an alternative to the tenzir/vast
image. The image is minimal in size and supports the same features as the
regular image, but does not support building additional plugins against it and
mounting in additional plugins.
Add CEF reader plugin
Section titled “Add CEF reader plugin”The cef import format allows for reading events in the Common Event Format
(CEF) via vast import cef < cef.log.
Add vast import arrow enabling ingestion of arrow IPC format
Section titled “Add vast import arrow enabling ingestion of arrow IPC format”Dec 12, 2022 · @dispanser · #2707
VAST now imports Arrow IPC data, which is the same format it already supports for export.
Add ‘pseudonymize’ pipeline operator
Section titled “Add ‘pseudonymize’ pipeline operator”Dec 10, 2022 · @Dakostu · #2719
The new pseudonymize pipeline operator pseudonymizes IP addresses in
user-specified fields.
Implement support for :string == /pattern/ queries
Section titled “Implement support for :string == /pattern/ queries”Dec 9, 2022 · @dominiklohmann · #2769
Queries of the forms :string == /pattern/, field == /pattern/, #type == /pattern/, and their respective negations now work as expected.
Eliminate shutdown lag from the signal monitor
Section titled “Eliminate shutdown lag from the signal monitor”The new /query endpoint for the experimental REST API allows users to receive
query data in multiple steps, as opposed to a oneshot export.
🔧 Changes
Section titled “🔧 Changes”Deprecate vast.pipeline-triggers
Section titled “Deprecate vast.pipeline-triggers”Mar 10, 2023 · @dominiklohmann · #3008
The vast.pipeline-triggers option is deprecated; while it continues to
work as-is, support for it will be removed in the next release. Use the
new inline import and export pipelines instead. They will return as more
generally applicable node ingress and egress pipelines in the future.
Replace ‘nil’ with ‘null
Section titled “Replace ‘nil’ with ‘null”Mar 10, 2023 · @Dakostu · #2999
The non-value literal in expressions has a new syntax: null replaces its old
representation nil. For example, the query x != nil is no longer valid; use
x != null instead.
Switch default TCP port to 5158
Section titled “Switch default TCP port to 5158”From now on VAST will use TCP port 5158 for its native inter process communication. This change avoids collisions from dynamic port allocation on Linux systems.
Remove the /export endpoint
Section titled “Remove the /export endpoint”Mar 7, 2023 · @Dakostu · #2990
The REST API does not contain the /export and /export/with-schemas
endpoints anymore. Any previous queries using those endpoints have to be sent
to the /query endpoint now.
Rename identity operator to pass
Section titled “Rename identity operator to pass”Mar 1, 2023 · @jachris · #2980
We renamed the identity operator to pass.
Make the map type inaccessible to users
Section titled “Make the map type inaccessible to users”Mar 1, 2023 · @dominiklohmann · #2976
The map type no longer exists: instead of map<T, U>, use the equivalent
list<record{ key: T, value: U }>.
Provide an Ansible role for VAST
Section titled “Provide an Ansible role for VAST”VAST now comes with a role definition for Ansible. You can find it directly in
the ansible subdirectory.
PRs 2922-2927
Section titled “PRs 2922-2927”Feb 8, 2023 · @dominiklohmann · #2922
We removed the frontend prototype bundled with the web plugin Some parts of the frontend that we have in development are designed to be closed-source, and it is easier to develop at the current development stage in a single repository that is not bound to the release process of VAST itself. An open-source version of the frontend may return in the future.
Use CAF streaming in export command
Section titled “Use CAF streaming in export command”The explore and pivot commands are now unavailable. They will be
reintroduced as pipeline operators in the future.
Update response format of the /export endpoint
Section titled “Update response format of the /export endpoint”For the experimental REST API, the result format of the /export endpoint
was modified: The num_events key was renamed to num-events, and the
version key was removed.
Align output of the Zeek TSV reader with schemas
Section titled “Align output of the Zeek TSV reader with schemas”Jan 28, 2023 · @dominiklohmann · #2887
The bundled Zeek schema no longer includes the _path field included in Zeek
JSON. Use #type == "zeek.foo" over _path == "foo" for querying data ingested
using vast import zeek-json.
Rename count, int, real, and addr to uint64, int64, double, and ip respectively
Section titled “Rename count, int, real, and addr to uint64, int64, double, and ip respectively”Jan 15, 2023 · @dominiklohmann · #2864
The builtin types count, int, real, and addr were renamed to uint64,
int64, double, and ip respectively. For backwards-compatibility, VAST
still supports parsing the old type tokens in schema files.
Move event distribution statistics to the catalog
Section titled “Move event distribution statistics to the catalog”Jan 12, 2023 · @dominiklohmann · #2852
The per-schema event distribution moved from index.statistics.layouts to
catalog.schemas, and additionally includes information about the import time
range and the number of partitions VAST knows for the schema. The number of
events per schema no longer includes events that are yet unpersisted.
Change boolean literals to true and false
Section titled “Change boolean literals to true and false”Jan 10, 2023 · @dominiklohmann · #2844
Boolean literals in expressions have a new syntax: true and false replace
the old representations T and F. For example, the query
suricata.alert.alerted == T is no longer valid; use suricata.alert.alerted == true instead.
Introduce a potpourri of smaller improvements
Section titled “Introduce a potpourri of smaller improvements”Jan 6, 2023 · @dominiklohmann · #2832
VAST now ignores the previously deprecated options vast.meta-index-fp-rate,
vast.catalog-fp-rate, vast.transforms and vast.transform-triggers.
Similarly, setting vast.store-backend to segment-store now results in an
error rather than a graceful fallback to the default store.
Introduce a potpourri of smaller improvements
Section titled “Introduce a potpourri of smaller improvements”Jan 6, 2023 · @dominiklohmann · #2832
Plugin names are now case-insensitive.
PRs 2807-2848
Section titled “PRs 2807-2848”Blocking imports now imply that ingested data gets persisted to disk before the
the vast import process exits.
Move taxonomy resolution to the catalog
Section titled “Move taxonomy resolution to the catalog”Dec 20, 2022 · @Dakostu · #2771
vast status does not work anymore with an embedded node (i.e., spawned with
the -N parameter).
PRs 2693-2923
Section titled “PRs 2693-2923”Dec 16, 2022 · @patszt · #2693
Building VAST now requires CAF 0.18.7. VAST supports setting advanced options
for CAF directly in its configuration file under the caf section. If you were
using any of these, compare them against the bundled vast.yaml.example file to
see if you need to make any changes. The change has (mostly positive)
performance and stability
implications
throughout VAST, especially in high-load scenarios.
Remove broker plugin
Section titled “Remove broker plugin”Dec 15, 2022 · @patszt · #2796
We removed the broker plugin that enabled direct Zeek 3.x log transfer to VAST. The plugin will return in the future rewritten for Zeek 5+.
PRs 2778-2797-2798
Section titled “PRs 2778-2797-2798”Dec 14, 2022 · @dominiklohmann · #2778
VAST no longer supports reading partitions created with VAST versions older than VAST v2.2. Since VAST v2.2, VAST continuously upgrades old partitions to the most recent internal format while running.
Merge the type-registry into the catalog
Section titled “Merge the type-registry into the catalog”Dec 10, 2022 · @Dakostu · #2745
vast status no longer shows type registry-related information. Instead, refer
to vast show for detailed type metadata information.
Add ‘pseudonymize’ pipeline operator
Section titled “Add ‘pseudonymize’ pipeline operator”Dec 10, 2022 · @Dakostu · #2719
OpenSSL is now a required dependency.
Remove the #field meta extractor
Section titled “Remove the #field meta extractor”Dec 9, 2022 · @dominiklohmann · #2776
The #field meta extractor no longer exists. Use X != null over #field == "X" to check for existence for the field X.
PRs 2769-2873
Section titled “PRs 2769-2873”Dec 9, 2022 · @dominiklohmann · #2769
The match operator ~, its negation !~, and the pattern type no longer
exist. Use queries of the forms lhs == /rhs/ and lhs != /rhs/ instead for
queries using regular expressions.
🐞 Bug Fixes
Section titled “🐞 Bug Fixes”Trigger new compaction runs immediately on error
Section titled “Trigger new compaction runs immediately on error”Mar 10, 2023 · @dominiklohmann · #3006
Compaction now retries immediately on failure instead of waiting for the configured scan interval to expire again.
Fix infinite recursion in the record algebra parser
Section titled “Fix infinite recursion in the record algebra parser”VAST no longer crashes when it encounters an invalid type expression in a schema.
Prevent query_processor from hanging when there are no candidate partitions
Section titled “Prevent query_processor from hanging when there are no candidate partitions”Feb 8, 2023 · @Dakostu · #2924
The VAST client will now terminate properly when using the count command with
a query which delivers zero results.
Fix rounding issue when printing duration types
Section titled “Fix rounding issue when printing duration types”We fixed incorrect printing of human-readable durations in some edge cases. E.g., the value 1.999s was rendered as 1.1s instead of the expected 2.0s. This bug affected the JSON and CSV export formats, and all durations printed in log messages or the status command.
Add a workaround to fix CAF OpenSSL options
Section titled “Add a workaround to fix CAF OpenSSL options”Options passed in the caf.openssl section in the configuration file or as
VAST_CAF__OPENSSL__* environment variables are no longer ignored.
Remove the transformer actor
Section titled “Remove the transformer actor”Feb 2, 2023 · @Dakostu · #2896
Pipelines that reduce the number of events do not prevent vast export
processes that have a max-events limit from terminating any more.
Align output of the Zeek TSV reader with schemas
Section titled “Align output of the Zeek TSV reader with schemas”Jan 28, 2023 · @dominiklohmann · #2887
The Zeek TSV reader now respects the schema files in the bundled zeek.schema
file, and produces data of the same schema as the Zeek JSON reader. E.g.,
instead of producing a top-level ip field id.orig_h, the reader now produces a
top-level record field id that contains the ip field orig_h, effectively
unflattening the data.
Fix some shutdown issues in the web plugin
Section titled “Fix some shutdown issues in the web plugin”The web plugin now reacts correctly to CTRL-C by stopping itself.
Run start commands asynchronously
Section titled “Run start commands asynchronously”The start commands specified with the vast.start.commands option are now run
aynchronously. This means that commands that block indefinitely will no longer
prevent execution of subsequent commands, and allow for correct signal handling.
Fix linux PID file check on startup
Section titled “Fix linux PID file check on startup”VAST no longer ignores existing PID lock files on Linux.
Introduce a potpourri of smaller improvements
Section titled “Introduce a potpourri of smaller improvements”Jan 6, 2023 · @dominiklohmann · #2832
VAST now shuts down instantly when metrics are enabled instead of being held alive for up to the duration of the telemetry interval (10 seconds).
Bump CAF to version 0.18.6
Section titled “Bump CAF to version 0.18.6”Dec 16, 2022 · @patszt · #2693
Attempting to connect with thousands of clients around the same time sometimes crashed the VAST server. This no longer occurs.
Infer non-config types in extend and replace operators
Section titled “Infer non-config types in extend and replace operators”Dec 10, 2022 · @dominiklohmann · #2768
The replace and extend pipeline operators wrongly inferred IP address,
subnet, pattern, and map values as strings. They are now inferred correctly. To
force a value to be inferred as a string, wrap it inside double quotes.