Download the release on GitHub.
Features
Section titled “Features”Summarize operator with pluggable aggregation functions
Section titled “Summarize operator with pluggable aggregation functions”The summarize operator supports three new aggregation functions: sample
takes the first value in every group, distinct filters out duplicate values,
and count yields the number of values.
By @dominiklohmann in #2417.
Implement a flush command
Section titled “Implement a flush command”The new flush command causes VAST to decommission all currently active
partitions, i.e., write all active partitions to disk immediately regardless of
their size or the active partition timeout. This is particularly useful for
testing, or when needing to guarantee in automated scripts that input is
available for operations that only work on persisted passive partitions. The
flush command returns only after all active partitions were flushed to disk.
By @dominiklohmann in #2396.
Introduce select / replace / extend operators
Section titled “Introduce select / replace / extend operators”The new extend pipeline operator allows for adding new fields with fixed
values to data.
By @dominiklohmann in #2423.
Run commands from scripts
Section titled “Run commands from scripts”The cloud execution commands (run-lambda and execute-command) now accept
scripts from file-like handles. To improve the usability of this feature, the
whole host file system is now mounted into the CLI container.
Support dropping entire schemas in drop operator
Section titled “Support dropping entire schemas in drop operator”The drop pipeline operator now drops entire schemas spcefied by name in the
schemas configuration key in addition to dropping fields by extractors in the
fields configuration key.
By @dominiklohmann in #2419.
Changes
Section titled “Changes”Add clean command
Section titled “Add clean command”An init command was added to vast-cloud to help getting out of inconsistent
Terraform states.
Implement Apache Parquet & Apache Feather V2 stores
Section titled “Implement Apache Parquet & Apache Feather V2 stores”Metrics for VAST’s store lookups now use the keys
{active,passive}-store.lookup.{runtime,hits}. The store type metadata field
now distinguishes between the various supported store types, e.g., parquet,
feather, or segment-store, rather than containing active or passive.
By @dominiklohmann in #2413.
Rename transform to pipeline
Section titled “Rename transform to pipeline”Transforms are now called pipelines. In your configuration, replace transform
with pipeline in all keys.
By @dispanser in #2429.
Introduce select / replace / extend operators
Section titled “Introduce select / replace / extend operators”The put pipeline operator is now called select, as we’ve abandoned plans to
integrate the functionality of replace into it.
The replace pipeline operator now supports multiple replacements in one
configuration, which aligns the behavior with other operators.
By @dominiklohmann in #2423.
Summarize operator with pluggable aggregation functions
Section titled “Summarize operator with pluggable aggregation functions”The summarize pipeline operator is now a builtin; the previously bundled
summarize plugin no longer exists. Aggregation functions in the summarize
operator are now plugins, which makes them easily extensible. The syntax of
summarize now supports specification of output field names, similar to SQL’s
AS in SELECT f(x) AS name.
The undocumented count pipeline operator no longer exists.
By @dominiklohmann in #2417.
Bug Fixes
Section titled “Bug Fixes”Make partition deletion resilient against oversize
Section titled “Make partition deletion resilient against oversize”VAST is now able to detect corrupt index files and will attempt to repair them on startup.
Make transform application transactional
Section titled “Make transform application transactional”We fixed a race condition when VAST crashed while applying a partition transform, leading to data duplication.
Properly indicate failure in the rebuild command
Section titled “Properly indicate failure in the rebuild command”The rebuild command no longer crashes on failure, and displays the encountered error instead.
By @dominiklohmann in #2466.
Fix missing options sometimes not causing an error
Section titled “Fix missing options sometimes not causing an error”Missing arguments for the --plugins, --plugin-dirs, and --schema-dirs
command line options no longer cause VAST to crash occasionally.
By @dominiklohmann in #2470.
Fix vast.export.json.omit-nulls for nested records
Section titled “Fix vast.export.json.omit-nulls for nested records”The JSON export with --omit-nulls now correctly handles nested records whose
first field is null instead of dropping them entirely.
By @dominiklohmann in #2447.
Render reals with at least one decimal place
Section titled “Render reals with at least one decimal place”VAST will export real values in JSON consistently with at least one decimal
place.