Enrichment is a major part of a security data lifecycle and can take on many
forms: adding GeoIP locations for all IP addresses in a log, attaching asset
inventory data via user or hostname lookups, or extending alerts with magic
score to bump it up the triaging queue. The goal is always to make the data more
actionable by providing a better ground for decision making.
This is the first part of series of blog posts on contextualization. We kick
things off by looking at how existing systems do enrichment. In the next blog
post, we introduce how we address this use case with pipeline-first mindset in
the Tenzir stack.
Elastic just released their new pipeline query language called
ES|QL. This is a conscious attempt to consolidate the language zoo in the
Elastic ecosystem
(queryDSL,
EQL,
KQL,
SQL,
Painless,
Canvas/Timelion).
Elastic said that they worked on this effort for over a year. The
documentation is still sparse, but we still tried to read between
the lines to understand what this new pipeline language has to offer.
Our Tenzir Query Language (TQL) is a pipeline language that works
by chaining operators into data flows. When we designed TQL, we specifically
studied Splunk's Search Processing Language (SPL), as it generally leaves
a positive impression for security analysts that are not data engineers. Our
goal was to take all the good things of SPL, but provide a more powerful
language without compromising simplicity. In this blog post, we explain how the
two languages differ using concrete threat hunting examples.
