Skip to main content

VAST v2.4.1

· 2 min read
Dominik Lohmann

VAST v2.4.1 improves the performance of queries when VAST is under high load, and significantly reduces the time to first result for queries with a low selectivity.

Reading Feather Files Incrementally

VAST's Feather store naïvely used the Feather reader from the Apache Arrow C++ library in its initial implementation. However, its API is rather limited: It does not support reading record batches incrementally. We've swapped this out with a more efficient implementation that does.

This is best explained visually:

Within the scope of a single Feather store file, a single query takes the same amount of time overall, but there exist two distinct advantages of this approach:

  1. The first result arrives much faster at the client.
  2. Stores do less work for cancelled queries.

One additional benefit that is not immediately obvious comes into play when queries arrives at multiple stores in parallel: disk reads are more evenly spread out now, making them less likely to overlap between stores. For deployments with slower I/O paths this can lead to a significant query performance improvement.

To verify and test this, we've created a VAST database with 300M Zeek events (33GB on disk) from a Corelight sensor. All tests were performed on a cold start of VAST, i.e., we stopped and started VAST after every repetition of each test.

We performed three tests:

  1. Export a single event (20 times)
  2. Export all events (20 times)
  3. Rebuild the entire database (3 times)

The results are astonishingly good:

(1)Avg. store load time55.1ms4.2ms13.1x
Time to first result/Total time19.8ms14.5ms1.4x
(2)Avg. store load time386.5ms7.3ms52.9x
Time to first result69.2ms25.4ms2.7x
Total time39.38s33.30s1.2x
(3)Avg. store load time480.3ms9.1ms52.7x
Total time210.5s198.0s1.1x

If you're using the Feather store backend (the default as of v2.4.0), you will see an immediate improvement with VAST v2.4.1. There are no other changes between the two releases.

Parquet Stores

VAST also offers an experimental Parquet store backend, for which we plan to make a similar improvement in a coming release.