Enrichment is a major part of a security data lifecycle and can take on many
forms: adding GeoIP locations for all IP addresses in a log, attaching asset
inventory data via user or hostname lookups, or extending alerts with magic
score to bump it up the triaging queue. The goal is always to make the data more
actionable by providing a better ground for decision making.
This is the first part of series of blog posts on contextualization. We kick
things off by looking at how existing systems do enrichment. In the next blog
post, we introduce how we address this use case with pipeline-first mindset in
the Tenzir stack.
The new velociraptor operator allows you to run
Velociraptor Query Language (VQL) expressions against a
Velociraptor server and process the results in a Tenzir
pipeline. You can also subscribe to matching artifacts in hunt flows over a
large fleet of assets, making endpoint telemetry collection and processing a
One thing we are observing is that organizations are actively seeking out
solutions to better manage their security data operations. Until recently, they
have been aggressively repurposing common data and observability tools. I
believe that this is a stop-gap measure because there was no alternative. But
now there is a growing ecosystem of security data operations tools to support
the modern security data stack. Ross Haleliuk's epic
lays this out at length.
In this article I am explaining the underlying design principles for developing
our own data pipeline engine, coming from the perspective of security teams that
are building out their detection and response architecture. These principles
emerged during design and implementation. Many times, we asked ourselves "what's
the right way of solving this problem?" We often went back to the drawing board
and started challenging existing approaches, such as what a data source is, or
what a connector should do. To our surprise, we found a coherent way to answer
these questions without having to make compromises. When things feel Just Right,
it is a good sign to have found the right solution for a particular problem.
What we are describing here are the lessons learned from studying other systems,
distilled as principles to follow for others.
Exciting times, Tenzir v4.3 is out! The headlining feature is Fluent
Bit support with the fluent-bitsource and
sink operators. Imagine you can use all Fluent Bit connectors
plus what Tenzir already offers. What a treat!