🚀 Features
Section titled “🚀 Features”Run commands from scripts
Section titled “Run commands from scripts”Jul 27, 2022 · @rdettai · #2446
The cloud execution commands (run-lambda and execute-command) now accept
scripts from file-like handles. To improve the usability of this feature, the
whole host file system is now mounted into the CLI container.
Introduce select / replace / extend operators
Section titled “Introduce select / replace / extend operators”Jul 12, 2022 · @dominiklohmann · #2423
The new extend pipeline operator allows for adding new fields with fixed
values to data.
Summarize operator with pluggable aggregation functions
Section titled “Summarize operator with pluggable aggregation functions”Jul 11, 2022 · @dominiklohmann · #2417
The summarize operator supports three new aggregation functions: sample
takes the first value in every group, distinct filters out duplicate values,
and count yields the number of values.
Support dropping entire schemas in drop operator
Section titled “Support dropping entire schemas in drop operator”Jul 11, 2022 · @dominiklohmann · #2419
The drop pipeline operator now drops entire schemas spcefied by name in the
schemas configuration key in addition to dropping fields by extractors in the
fields configuration key.
Implement a flush command
Section titled “Implement a flush command”Jul 8, 2022 · @dominiklohmann · #2396
The new flush command causes VAST to decommission all currently active
partitions, i.e., write all active partitions to disk immediately regardless of
their size or the active partition timeout. This is particularly useful for
testing, or when needing to guarantee in automated scripts that input is
available for operations that only work on persisted passive partitions. The
flush command returns only after all active partitions were flushed to disk.
🔧 Changes
Section titled “🔧 Changes”Add clean command
Section titled “Add clean command”Jul 28, 2022 · @rdettai · #2435
An init command was added to vast-cloud to help getting out of inconsistent
Terraform states.
Rename transform to pipeline
Section titled “Rename transform to pipeline”Jul 16, 2022 · @dispanser · #2429
Transforms are now called pipelines. In your configuration, replace transform
with pipeline in all keys.
Implement Apache Parquet & Apache Feather V2 stores
Section titled “Implement Apache Parquet & Apache Feather V2 stores”Jul 14, 2022 · @dominiklohmann · #2413
Metrics for VAST’s store lookups now use the keys
{active,passive}-store.lookup.{runtime,hits}. The store type metadata field
now distinguishes between the various supported store types, e.g., parquet,
feather, or segment-store, rather than containing active or passive.
Introduce select / replace / extend operators
Section titled “Introduce select / replace / extend operators”Jul 12, 2022 · @dominiklohmann · #2423
The put pipeline operator is now called select, as we’ve abandoned plans to
integrate the functionality of replace into it.
The replace pipeline operator now supports multiple replacements in one
configuration, which aligns the behavior with other operators.
Summarize operator with pluggable aggregation functions
Section titled “Summarize operator with pluggable aggregation functions”Jul 11, 2022 · @dominiklohmann · #2417
The summarize pipeline operator is now a builtin; the previously bundled
summarize plugin no longer exists. Aggregation functions in the summarize
operator are now plugins, which makes them easily extensible. The syntax of
summarize now supports specification of output field names, similar to SQL’s
AS in SELECT f(x) AS name.
The undocumented count pipeline operator no longer exists.
🐞 Bug Fixes
Section titled “🐞 Bug Fixes”Make transform application transactional
Section titled “Make transform application transactional”We fixed a race condition when VAST crashed while applying a partition transform, leading to data duplication.
Fix missing options sometimes not causing an error
Section titled “Fix missing options sometimes not causing an error”Jul 27, 2022 · @dominiklohmann · #2470
Missing arguments for the --plugins, --plugin-dirs, and --schema-dirs
command line options no longer cause VAST to crash occasionally.
Properly indicate failure in the rebuild command
Section titled “Properly indicate failure in the rebuild command”Jul 27, 2022 · @dominiklohmann · #2466
The rebuild command no longer crashes on failure, and displays the encountered error instead.
Make partition deletion resilient against oversize
Section titled “Make partition deletion resilient against oversize”VAST is now able to detect corrupt index files and will attempt to repair them on startup.
Fix vast.export.json.omit-nulls for nested records
Section titled “Fix vast.export.json.omit-nulls for nested records”Jul 20, 2022 · @dominiklohmann · #2447
The JSON export with --omit-nulls now correctly handles nested records whose
first field is null instead of dropping them entirely.
Render reals with at least one decimal place
Section titled “Render reals with at least one decimal place”Jul 11, 2022 · @patszt · #2393
VAST will export real values in JSON consistently with at least one decimal
place.