Download the release on GitHub.
Features
Section titled “Features”Add vast import arrow enabling ingestion of arrow IPC format
Section titled “Add vast import arrow enabling ingestion of arrow IPC format”VAST now imports Arrow IPC data, which is the same format it already supports for export.
By @dispanser in #2707.
Add CORS preflight request handling
Section titled “Add CORS preflight request handling”The experimental web frontend now correctly responds to CORS preflight requests.
To configure CORS behavior, the new vast.web.cors-allowed-origin config option
can be used.
Support pattern case insensitivity in Sigma plugin
Section titled “Support pattern case insensitivity in Sigma plugin”The sigma plugin now treats Sigma strings as case-insensitive patterns during
the transpilation process.
Make it easy to create docker images with Nix
Section titled “Make it easy to create docker images with Nix”We now offer a tenzir/vast-slim image as an alternative to the tenzir/vast
image. The image is minimal in size and supports the same features as the
regular image, but does not support building additional plugins against it and
mounting in additional plugins.
Add ‘pseudonymize’ pipeline operator
Section titled “Add ‘pseudonymize’ pipeline operator”The new pseudonymize pipeline operator pseudonymizes IP addresses in
user-specified fields.
Implement support for :string == /pattern/ queries
Section titled “Implement support for :string == /pattern/ queries”Queries of the forms :string == /pattern/, field == /pattern/, #type == /pattern/, and their respective negations now work as expected.
By @dominiklohmann in #2769.
Add options to omit empty values when exporting as JSON
Section titled “Add options to omit empty values when exporting as JSON”The JSON export format gained the options --omit-empty-records,
--omit-empty-lists, and --omit-empty-maps, which cause empty records, lists,
and maps not to be rendered respectively. The options may be combined together
with the existing --omit-nulls option. Use --omit-empty to set all four
flags at once.
By @dominiklohmann in #2856.
Add ‘pipeline’ parameter and schematized format to export endpoint
Section titled “Add ‘pipeline’ parameter and schematized format to export endpoint”The /export family of endpoints now accepts an optional pipeline
parameter to specify an ad-hoc pipeline that should be applied to
the exported data.
Support case insensitivity in patterns
Section titled “Support case insensitivity in patterns”Patterns now support case insensitivity by adding i to the pattern string,
e.g. /^\w{3}$/i.
Add a new /openapi.json endpoint to the web plugin
Section titled “Add a new /openapi.json endpoint to the web plugin”The experimental web plugin now serves its own API specification at the new ‘/openapi.json’ endpoint.
Eliminate shutdown lag from the signal monitor
Section titled “Eliminate shutdown lag from the signal monitor”The new /query endpoint for the experimental REST API allows users to receive
query data in multiple steps, as opposed to a oneshot export.
Implement head and taste operators
Section titled “Implement head and taste operators”The new head and taste operators limit results to the specified number of
events. The head operator applies this limit for all events, and the taste
operator applies it per schema. Both operators take the limit as an optional
argument, with the default value being 10.
By @dominiklohmann in #2891.
PRs 2877-2904-2907
Section titled “PRs 2877-2904-2907”The export and import commands now support an optional pipeline string
that allows for chaining pipeline operators together and executing such a
pipeline on outgoing and incoming data. This feature is experimental and the
syntax is subject to change without notice. New operators are only available in
the new pipeline syntax, and the old YAML syntax is deprecated.
Add extractor predicates
Section titled “Add extractor predicates”Extractors such as x and :T can now expand to the predicates x != null
and :T != null, respectively.
Implement a retry mechanism for VAST clients failing to connect to the server
Section titled “Implement a retry mechanism for VAST clients failing to connect to the server”We changed VAST client processes to attempt connecting to a VAST server multiple
times until the configured connection timeout (vast.connection-timeout,
defaults to 5 minutes) runs out. A fixed delay between connection attempts
(vast.connection-retry-delay, defaults to 3 seconds) ensures that clients to
not stress the server too much. Set the connection timeout to zero to let VAST
client attempt connecting indefinitely, and the delay to zero to disable the
retry mechanism.
Install Python bindings along with VAST
Section titled “Install Python bindings along with VAST”VAST installations and packages now include Python bindings in a site-package
under <install-prefix>/lib/python*/site-packages/vast.
By @dominiklohmann in #2636.
Add CEF reader plugin
Section titled “Add CEF reader plugin”The cef import format allows for reading events in the Common Event Format
(CEF) via vast import cef < cef.log.
Changes
Section titled “Changes”Add ‘pseudonymize’ pipeline operator
Section titled “Add ‘pseudonymize’ pipeline operator”OpenSSL is now a required dependency.
Remove broker plugin
Section titled “Remove broker plugin”We removed the broker plugin that enabled direct Zeek 3.x log transfer to VAST. The plugin will return in the future rewritten for Zeek 5+.
PRs 2769-2873
Section titled “PRs 2769-2873”The match operator ~, its negation !~, and the pattern type no longer
exist. Use queries of the forms lhs == /rhs/ and lhs != /rhs/ instead for
queries using regular expressions.
By @dominiklohmann in #2769.
Replace ‘nil’ with ‘null’
Section titled “Replace ‘nil’ with ‘null’”The non-value literal in expressions has a new syntax: null replaces its old
representation nil. For example, the query x != nil is no longer valid; use
x != null instead.
Deprecate vast.pipeline-triggers
Section titled “Deprecate vast.pipeline-triggers”The vast.pipeline-triggers option is deprecated; while it continues to
work as-is, support for it will be removed in the next release. Use the
new inline import and export pipelines instead. They will return as more
generally applicable node ingress and egress pipelines in the future.
By @dominiklohmann in #3008.
Remove the #field meta extractor
Section titled “Remove the #field meta extractor”The #field meta extractor no longer exists. Use X != null over #field == "X" to check for existence for the field X.
By @dominiklohmann in #2776.
Rename count, int, real, and addr to uint64, int64, double, and ip respectively
Section titled “Rename count, int, real, and addr to uint64, int64, double, and ip respectively”The builtin types count, int, real, and addr were renamed to uint64,
int64, double, and ip respectively. For backwards-compatibility, VAST
still supports parsing the old type tokens in schema files.
By @dominiklohmann in #2864.
Use CAF streaming in export command
Section titled “Use CAF streaming in export command”The explore and pivot commands are now unavailable. They will be
reintroduced as pipeline operators in the future.
Introduce a potpourri of smaller improvements
Section titled “Introduce a potpourri of smaller improvements”Plugin names are now case-insensitive.
By @dominiklohmann in #2832.
Introduce a potpourri of smaller improvements
Section titled “Introduce a potpourri of smaller improvements”VAST now ignores the previously deprecated options vast.meta-index-fp-rate,
vast.catalog-fp-rate, vast.transforms and vast.transform-triggers.
Similarly, setting vast.store-backend to segment-store now results in an
error rather than a graceful fallback to the default store.
By @dominiklohmann in #2832.
Provide an Ansible role for VAST
Section titled “Provide an Ansible role for VAST”VAST now comes with a role definition for Ansible. You can find it directly in
the ansible subdirectory.
Rename identity operator to pass
Section titled “Rename identity operator to pass”We renamed the identity operator to pass.
PRs 2922-2927
Section titled “PRs 2922-2927”We removed the frontend prototype bundled with the web plugin Some parts of the frontend that we have in development are designed to be closed-source, and it is easier to develop at the current development stage in a single repository that is not bound to the release process of VAST itself. An open-source version of the frontend may return in the future.
By @dominiklohmann in #2922.
Update response format of the /export endpoint
Section titled “Update response format of the /export endpoint”For the experimental REST API, the result format of the /export endpoint
was modified: The num_events key was renamed to num-events, and the
version key was removed.
Change boolean literals to true and false
Section titled “Change boolean literals to true and false”Boolean literals in expressions have a new syntax: true and false replace
the old representations T and F. For example, the query
suricata.alert.alerted == T is no longer valid; use suricata.alert.alerted == true instead.
By @dominiklohmann in #2844.
Make the map type inaccessible to users
Section titled “Make the map type inaccessible to users”The map type no longer exists: instead of map<T, U>, use the equivalent
list<record{ key: T, value: U }>.
By @dominiklohmann in #2976.
Switch default TCP port to 5158
Section titled “Switch default TCP port to 5158”From now on VAST will use TCP port 5158 for its native inter process communication. This change avoids collisions from dynamic port allocation on Linux systems.
PRs 2807-2848
Section titled “PRs 2807-2848”Blocking imports now imply that ingested data gets persisted to disk before the
the vast import process exits.
Remove the /export endpoint
Section titled “Remove the /export endpoint”The REST API does not contain the /export and /export/with-schemas
endpoints anymore. Any previous queries using those endpoints have to be sent
to the /query endpoint now.
Merge the type-registry into the catalog
Section titled “Merge the type-registry into the catalog”vast status no longer shows type registry-related information. Instead, refer
to vast show for detailed type metadata information.
PRs 2693-2923
Section titled “PRs 2693-2923”Building VAST now requires CAF 0.18.7. VAST supports setting advanced options
for CAF directly in its configuration file under the caf section. If you were
using any of these, compare them against the bundled vast.yaml.example file to
see if you need to make any changes. The change has (mostly positive)
performance and stability
implications
throughout VAST, especially in high-load scenarios.
Align output of the Zeek TSV reader with schemas
Section titled “Align output of the Zeek TSV reader with schemas”The bundled Zeek schema no longer includes the _path field included in Zeek
JSON. Use #type == "zeek.foo" over _path == "foo" for querying data ingested
using vast import zeek-json.
By @dominiklohmann in #2887.
PRs 2778-2797-2798
Section titled “PRs 2778-2797-2798”VAST no longer supports reading partitions created with VAST versions older than VAST v2.2. Since VAST v2.2, VAST continuously upgrades old partitions to the most recent internal format while running.
By @dominiklohmann in #2778.
Move event distribution statistics to the catalog
Section titled “Move event distribution statistics to the catalog”The per-schema event distribution moved from index.statistics.layouts to
catalog.schemas, and additionally includes information about the import time
range and the number of partitions VAST knows for the schema. The number of
events per schema no longer includes events that are yet unpersisted.
By @dominiklohmann in #2852.
Move taxonomy resolution to the catalog
Section titled “Move taxonomy resolution to the catalog”vast status does not work anymore with an embedded node (i.e., spawned with
the -N parameter).
Bug Fixes
Section titled “Bug Fixes”Fix rounding issue when printing duration types
Section titled “Fix rounding issue when printing duration types”We fixed incorrect printing of human-readable durations in some edge cases. E.g., the value 1.999s was rendered as 1.1s instead of the expected 2.0s. This bug affected the JSON and CSV export formats, and all durations printed in log messages or the status command.
Fix linux PID file check on startup
Section titled “Fix linux PID file check on startup”VAST no longer ignores existing PID lock files on Linux.
Add a workaround to fix CAF OpenSSL options
Section titled “Add a workaround to fix CAF OpenSSL options”Options passed in the caf.openssl section in the configuration file or as
VAST_CAF__OPENSSL__* environment variables are no longer ignored.
Align output of the Zeek TSV reader with schemas
Section titled “Align output of the Zeek TSV reader with schemas”The Zeek TSV reader now respects the schema files in the bundled zeek.schema
file, and produces data of the same schema as the Zeek JSON reader. E.g.,
instead of producing a top-level ip field id.orig_h, the reader now produces a
top-level record field id that contains the ip field orig_h, effectively
unflattening the data.
By @dominiklohmann in #2887.
Prevent query_processor from hanging when there are no candidate partitions
Section titled “Prevent query_processor from hanging when there are no candidate partitions”The VAST client will now terminate properly when using the count command with
a query which delivers zero results.
Run start commands asynchronously
Section titled “Run start commands asynchronously”The start commands specified with the vast.start.commands option are now run
aynchronously. This means that commands that block indefinitely will no longer
prevent execution of subsequent commands, and allow for correct signal handling.
Bump CAF to version 0.18.6
Section titled “Bump CAF to version 0.18.6”Attempting to connect with thousands of clients around the same time sometimes crashed the VAST server. This no longer occurs.
Remove the transformer actor
Section titled “Remove the transformer actor”Pipelines that reduce the number of events do not prevent vast export
processes that have a max-events limit from terminating any more.
Infer non-config types in extend and replace operators
Section titled “Infer non-config types in extend and replace operators”The replace and extend pipeline operators wrongly inferred IP address,
subnet, pattern, and map values as strings. They are now inferred correctly. To
force a value to be inferred as a string, wrap it inside double quotes.
By @dominiklohmann in #2768.
Trigger new compaction runs immediately on error
Section titled “Trigger new compaction runs immediately on error”Compaction now retries immediately on failure instead of waiting for the configured scan interval to expire again.
By @dominiklohmann in #3006.
Fix some shutdown issues in the web plugin
Section titled “Fix some shutdown issues in the web plugin”The web plugin now reacts correctly to CTRL-C by stopping itself.
Fix infinite recursion in the record algebra parser
Section titled “Fix infinite recursion in the record algebra parser”VAST no longer crashes when it encounters an invalid type expression in a schema.
Introduce a potpourri of smaller improvements
Section titled “Introduce a potpourri of smaller improvements”VAST now shuts down instantly when metrics are enabled instead of being held alive for up to the duration of the telemetry interval (10 seconds).
By @dominiklohmann in #2832.