Metrics

VAST keeps detailed track of system metrics that reflect runtime state, such ingestion performance, query latencies, and resource usage. The metrics data behaves like ordinary user data: querying it works the same way.

Internally, every component sends its metrics to a central accountant actor, which in turn relays the metrics feed back to the regular ingestion path.

Usage

The accountant is enabled by default and waits for metrics reports from other components. It represents its data as the vast.metrics event layout and can thus be queried like any other in VAST.

For example, the following query searches for all metrics events with the key pcap-reader.recv, which keeps track of a delta of received packets on all interfaces VAST listens on. With JSON as export format, we can use jq to calculate the sum of these values to get the total amount of packets seen on all interfaces.

vast export json '#type == "vast.metrics" && key == "pcap-reader.recv"' |
jq -s 'map(.value | tonumber) | add'

You can disable the accountant and thereby metrics collection. Either when starting a node (vast --disable-metrics start) or the vast.conf:

system {
disable-metrics = true
}
note

Collecting metrics is optional and incurs minimal overhead. Disabling the accountant should only be done when disk space is scarce or every last bit of performance needs to be made available to other components of VAST.

Data Representation

The accountant generates events of type vast.metrics, which has the following schema:

type vast.metrics = record {
ts: time #timestamp,
nodeid: string,
aid: count,
actor_name: string,
key: string,
value: string,
}

Available Keys

KeyDescription
exporter.hitsTODO
exporter.processedTODO
exporter.resultsTODO
exporter.shippedTODO
exporter.selectivityTODO
exporter.runtimeTODO
exporter.hits.firstTODO
exporter.hits.arrivedTODO
exporter.hits.countTODO
exporter.hits.runtimeTODO
source.startTODO
importerTODO
node_throughputTODO
TODO

The above list is incomplete. We will gradually add keys and their descriptions over time. Stay tuned.