Download the release on GitHub.
Features
Section titled “Features”PRs 1407-1487-1490
Section titled “PRs 1407-1487-1490”The schema language now supports 4 operations on record types: + combines the
fields of 2 records into a new record. <+ and +> are variations of + that
give precedence to the left and right operand respectively. - creates a record
with the field specified as its right operand removed.
Support nested records in the Arrow Builder
Section titled “Support nested records in the Arrow Builder”VAST now supports nested records in Arrow table slices and in the JSON import,
e.g., data of type list<record<name: string, age: count>. While nested record
fields are not yet queryable, ingesting such data will no longer cause VAST to
crash. MessagePack table slices don’t support records in lists yet.
By @dominiklohmann in #1429.
Changes
Section titled “Changes”Move zeek-to-vast from tenzir/vast to tenzir/zeek-vast
Section titled “Move zeek-to-vast from tenzir/vast to tenzir/zeek-vast”The zeek-to-vast utility was moved to the tenzir/zeek-vast repository. All options related to zeek-to-vast and the bundled Broker submodule were removed.
By @dominiklohmann in #1435.
PRs 1408-satta
Section titled “PRs 1408-satta”VAST now ships with schema record types for Suricata’s mqtt and anomaly
event types.
Deprecate the vast.no-default-schema option
Section titled “Deprecate the vast.no-default-schema option”The option vast.no-default-schema is deprecated, as it is no longer needed to
override types from bundled schemas.
By @dominiklohmann in #1409.
PRs 1445-1452
Section titled “PRs 1445-1452”Plugins can now be linked statically against VAST. A new VASTRegisterPlugin
CMake function enables easy setup of the build scaffolding required for plugins.
Configure with --with-static-plugins or build a static binary to link all
plugins built alongside VAST statically. All plugin build scaffoldings must be
adapted, older plugins do no longer work.
By @dominiklohmann in #1445.
Rename the attribute_extractor to meta_extractor
Section titled “Rename the attribute_extractor to meta_extractor”The previously deprecated #timestamp extractor has been removed from the query
language entirely. Use :timestamp instead.
Establish subtyping relationships for type extractors
Section titled “Establish subtyping relationships for type extractors”The type extractor in the expression language now works with type aliases. For
example, given the type definition for port from the base schema type port = count, a search for :count will also consider fields of type port.
Change the default batch size to 1024
Section titled “Change the default batch size to 1024”The default size of table slices (event batches) that is created from vast import processes has been changed from 1,000 to 1,024.
Prune expressions for the meta index lookup
Section titled “Prune expressions for the meta index lookup”Query latency for expressions that contain concept names has improved substantially. For DB sizes in the TB region, and with a large variety of event types, queries with a high selectivity experience speedups of up to 5x.
Bug Fixes
Section titled “Bug Fixes”Remove erased partitions from the meta index
Section titled “Remove erased partitions from the meta index”The disk monitor now correctly erases partition synopses from the meta index.
Fix printing of non-null intrusive pointers
Section titled “Fix printing of non-null intrusive pointers”Some non-null pointers were incorrectly rendered as *nullptr in log messages.
Don’t allow field extractors to match field name suffixes
Section titled “Don’t allow field extractors to match field name suffixes”A query for a field or field name suffix that matches multiple fields of different types would erroneously return no results.
Fix possibly unhandled exception in disk monitor
Section titled “Fix possibly unhandled exception in disk monitor”VAST no longer crashes when the disk monitor tries to calculate the size of the database while files are being deleted. Instead, it will retry after the configured scan interval.
By @dominiklohmann in #1458.
Make the disk-monitor a singleton component
Section titled “Make the disk-monitor a singleton component”Enabling the disk budget feature no longer prevents the server process from exiting after it was stopped.
Use non-throwing std::filesystem functions in the type registry
Section titled “Use non-throwing std::filesystem functions in the type registry”Insufficient permissions for one of the paths in the schema-dirs option would
lead to a crash in vast start.
PRs 1473-1485
Section titled “PRs 1473-1485”A race condition during server shutdown could lead to an invariant violation, resulting in a firing assertion. Streamlining the shutdown logic resolved the issue.
Report metrics while idle
Section titled “Report metrics while idle”The archive, index, source, and sink components now report metrics when idle instead of omitting them entirely. This allows for distinguishing between idle and not running components from the metrics.
By @dominiklohmann in #1451.
Fix :timestamp queries for old data
Section titled “Fix :timestamp queries for old data”Data that was ingested before the deprecation of the #timestamp attribute
wasn’t exported correctly with newer versions. This is now corrected.
Accept numbers in place of strings in JSON
Section titled “Accept numbers in place of strings in JSON”The JSON parser now accepts data with numerical or boolean values in fields that expect strings according to the schema. VAST converts these values into string representations.