Aging

Experimental Feature

This is an experimental feature: the API is subject to change and robustness is not yet comperable to production-grade features.

The aging feature enables periodic deletion of events based on user-specified queries. When would you use this?

  • The retention policy requires to store data only up to N months.
  • The amount of new data exceeds your storage budget in the foreseeable future.
  • Sensitive data may enter the system but should not be recorded.

Usage

To use the aging feature, you need to edit the configuration file and provide an frequency and query for erasing the data. Alternatively, the two options can be supplied to the vast start command:

vast --aging-frequency=<timespan> --aging-query=<query> start

Here is an example of the configuration-based approach via vast.conf:

system {
; Run the aging algorithm twice every day.
aging-frequency = "12 hours"
; Remove all events that have a timestamp associated with them that's older
; than 7 days.
aging-query = "#timestamp < 7 days ago"
}
note

We marked the feature as experimental, because aging only removes events from the archive and not from the index. Removing data from the index is non-trivial, because the index is partially lossy and may require re-indexing of the base data to remain intact. We are already working supporting deletion of data from the index.

As a side effect, index statistics from the vast status command are no longer reliable after an aging cycle. This is why the feature