VAST is now Tenzir. This blog post describes what changed when we renamed the project.
Visibility Across Space and Time is now Tenzir
After 5 years of developing two identities, the VAST project and Tenzir the company, we decided to streamline our efforts and rename VAST to Tenzir.
VAST v3.1
VAST v3.1 is out. This is a small checkpointing release that brings a few new changes and fixes.
VAST v3.0
VAST v3.0 is out. This release brings some major updates to the the VAST language, making it easy to write down dataflow pipelines that filter, reshape, aggregate, and enrich security event data. Think of VAST as security data pipelines plus open storage engine.
From Slack to Discord
The New REST API
As of v2.4 VAST ships with a new web
plugin that
provides a REST API. The API documentation describes the
available endpoints also provides an
OpenAPI spec for download. This
blog post shows how we built the API and what you can do with it.
Parquet & Feather: Data Engineering Woes
Apache Arrow and Apache Parquet have become the de-facto columnar formats for in-memory and on-disk representations when it comes to structured data. Both are strong together, as they provide data interoperability and foster a diverse ecosystem of data tools. But how well do they actually work together from an engineering perspective?
VAST v2.4.1
VAST v2.4.1 improves the performance of queries when VAST is under high load, and significantly reduces the time to first result for queries with a low selectivity.
VAST v2.4
VAST v2.4 completes the switch to open storage formats, and includes an early peek at three upcoming features for VAST: A web plugin with a REST API and an integrated frontend user interface, Docker Compose configuration files for getting started with VAST faster and showing how to integrate VAST into your SOC, and new Python bindings that will make writing integrations easier and allow for using VAST with your data science libraries, like Pandas.
Parquet & Feather: Writing Security Telemetry
VAST v2.3.1
VAST v2.3.1 is now available. This small bugfix release
addresses an issue where compaction would hang if encountering
invalid partitions that were produced by older versions of VAST when a large
max-partition-size
was set in combination with badly compressible input data.
Parquet & Feather: Enabling Open Investigations
Apache Parquet is the common denominator for structured data at rest. The data science ecosystem has long appreciated this. But infosec? Why should you care about Parquet when building a threat detection and investigation platform? In this blog post series we share our opinionated view on this question. In the next three blog posts, we
- describe how VAST uses Parquet and its little brother Feather
- benchmark the two formats against each other for typical workloads
- share our experience with all the engineering gotchas we encountered along the way
A Git Retrospective
The VAST project is roughly a decade old. But what happened over the last 10 years? This blog post looks back over time through the lens of the git merge commits.
Why merge commits? Because they represent a unit of completed contribution.
Feature work takes place in dedicated branches, with the merge to the main
branch sealing the deal. Some feature branches have just one commit, whereas
others dozens. The distribution is not uniform. As of 6f9c84198
on Sep 2,
2022, there are a total of 13,066 commits, with 2,334 being merges (17.9%).
We’ll take a deeper look at the merge commits.
Public Roadmap and Open RFCs
We are happy to announce that we have published our engineering roadmap along with an RFC process to actively participate in shaping upcoming topics. This blog post explains why and how we did it.
VAST v2.3
VAST v2.3 is now available, which introduces an automatic data defragmentation capability.
Richer Typing in Sigma
VAST's Sigma frontend now supports more modifiers. In the Sigma language, modifiers transform predicates in various ways, e.g., to apply a function over a value or to change the operator of a predicate. Modifiers are the customization point to enhance expressiveness of query operations.
The new pySigma effort, which will eventually replace the
now-considered-legacy sigma project, comes with new modifiers as well.
Most notably, lt
, lte
, gt
, gte
provide comparisons over value domains
with a total ordering, e.g., numbers: x >= 42
. In addition, the cidr
modifier interprets a value as subnet, e.g., 10.0.0.0/8
. Richer typing!
VAST v2.2
We released VAST v2.2 🙌! Transforms now have a new name: pipelines. The summarize operator also underwent a facelift, making aggregation functions pluggable and allowing for assigning names to output fields.
VAST v2.1
VAST v2.1 is out! This release comes with a particular focus on performance and reducing the size of VAST databases. It brings a new utility for optimizing databases in production, allowing existing deployments to take full advantage of the improvements after upgrading.
Apache Arrow as Platform for Security Data Engineering
VAST bets on Apache Arrow as the open interface to structured data. By "bet," we mean that VAST does not work without Arrow. And we are not alone. Influx's IOx, DataDog's Husky, Anyscale's Ray, TensorBase, and others committed themselves to making Arrow a corner stone of their system architecture. For us, Arrow was not always a required dependency. We shifted to a tighter integration over the years as the Arrow ecosystem matured. In this blog post we explain our journey of becoming an Arrow-native engine.
VAST v2.0
Dear community, we are excited to announce VAST v2.0, bringing faster execution of bulk-submitted queries, improved tunability of index structures, and new configurability through environment variables.