VAST v3.0 is out. This release brings some major updates
to the the VAST language, making it easy to write down dataflow pipelines that
filter, reshape, aggregate, and enrich security event data. Think of VAST as
security data pipelines plus open storage engine.
Starting with v3.0, VAST introduces a new way to write pipelines, with a syntax
similar to splunk, Kusto,
PRQL, and Zed. Previously,
VAST only supported a YAML-like definition of pipelines in configuration files
to deploy them statically during import, export, or use them during compaction.
The new syntax resembles the well-known Unix paradigm of command chaining. The
difference to Unix pipelines is that VAST exchanges structured data between
operators. The vast export and vast import commands now accept such a
pipeline as a string argument. Refer to the pipelines
documentation for more details on how to use the new pipeline
syntax.
Pipeline YAML Syntax Deprecation
This release introduces a transitional period from YAML-style to textual
pipelines. The old YAML syntax for pipelines will be deprecated and removed
altogether in a future version. The new pipeline operators head and
taste have no YAML equivalent.
We've made some breaking changes to the the VAST language that we've wanted to
do for a long time. Here's a summary:
We removed the term VASTQL: The VAST Query Language is now simply
the VAST language, and "VAST" will supersede the "VASTQL" abbreviation.
Several built-in types have a new name:
int → int64
count → uint64
real → double
addr → ip
The old names are still supported for the time being, but trigger a
warning on startup. We will remove support for the old names in a future
release.
The match operator ~ and its negated form !~ no longer exist. Use ==
and != instead to perform searches with regular expressions, e.g., url ==
/^https?.*/. Such queries now work for all string fields in addition to the
previously supported #type meta extractor.
We removed the #field meta extractor. That is, queries of the form #field
== "some.field.name" no longer work. Use some.field.name != null or the
new short form some.field.name to check for field existence moving forward.
We renamed the boolean literal values T and F to true and false,
respectively. For example the query suricata.alert.alerted == T is no
longer valid; use suricata.alert.alerted == true instead.
We renamed the non-value literal value nil to null. For example the
query x != nil is no longer valid; use x != null instead.
The map type no longer exists: Instead of map<T, U>, use the equivalent
list<record{ key: T, value: U }>.
Our goal is for these changes to make the query language feel more natural to
our users. We've got big plans on how to extend it—and this felt very
much necessary as a preparatory step to making the language more useful.
VAST now supports searching with regular expressions. For example, let's say you
are looking for all events that contain a GUID surrounded by braces whose third
and fourth section are 5ec2-7015:
Case-Insensitive Patterns
In addition to writing /pattern/, you can specify a regular expression that
ignores the casing of characters via /pattern/i. The /i flag is currently
the only support pattern modifier.
The vast show schemas command makes it easy to see the structure of events in
the database at a glance.
Filter Schemas
The vast show schemas command supports filtering not just by the exact name of
a schema, but also by the module name. E.g., vast show schemas zeek will print
a list of all schemas in the Zeek module that the VAST server holds data for.
This release includes a new reader plugin for the Common Event Format
(CEF), a text-based event format that originally stems from ArcSight. This
line-based format consists of up to 8 pipe-separated fields, with the last field
being an optional list of key-value pairs:
CEF:Version|Device Vendor|Device Product|Device Version|Device Event Class ID|Name|Severity|[Extension]
VAST's CEF plugin supports parsing such lines using the cef format:
vast import cef < cef.log
VAST translates the extension field to a nested record, where the key-value
pairs of the extensions map to record fields. Here is an example of the above
event:
Syslog Header
Sometimes CEF is prefixed with a syslog header. VAST currently only supports the
"raw" form without the syslog header. We are working on support for composable
generic formats, e.g., syslog, where the message can basically be any other
existing format.
This VAST release contains a fair amount of other changes and interesting
improvements. As always, the changelog contains a complete list of
user-facing changes since the last release.
The vast export json command gained new options in addition to the already
existing --omit-nulls: Pass --omit-empty-records, --omit-empty-lists,
or --omit-empty-maps to cause VAST not to display empty records, lists, or
maps respectively.
The flag --omit-empty empty combines the three new options and --omit-nulls,
essentially causing VAST not to render empty values at all. To set these options
globally, add the following to your vast.yaml configuration file:
VAST processes now shut down faster, which especially improves the performance
of the vast import and vast export commands for small amounts of data
ingested or quickly finishing queries.
To quantify this, we've created a database with nearly 300M Zeek events, and ran
an export of a single event with both VAST v2.4.1 and VAST v3.0 repeatedly.
VAST clients may now be started before the VAST server: Client processes now
attempt to connect to server processes repeatedly until the configured
connection timeout expires.
We found this to generally improve reliability of services with multiple VAST
clients, for which we often encountered problems with VAST clients being unable
to connect to a VAST server when started before or immediately after the VAST
server.
Additionally, we've fixed a bug that caused VAST to crash when thousands of
clients attempted to connect at around the same time.
The new tenzir/vast-slim Docker image is an alternative to the existing
tenzir/vast Docker image that comes in at just under 40 MB in size—less than a
third than the regular image, making it even quicker to get started with VAST.
Extractors can now be used where predicates are expected to test for the
existance of a field or type. For example, x and :T expand to x != null
and :T != null, respectively. This pairs nicely with the already existing
short forms for values, e.g., "foo" expands to :string == "foo.