TQL2 Migration
This page answers the most frequently asked questions about TQL2.
What is TQL2?
TQL2 is the next generation of the Tenzir Query Language for writing pipelines.
How do I use TQL2?
Start your pipeline with a // tql2 comment to opt into using the new language.
This is supported for Tenzir v4.19 or newer.
Why create an all-new language?
TQL1 has grown historically. Over time, we identified the following shortcomings that TQL2 fixes:
- Lack of composable expressions, in particular functions and arithmetic operations.
- Absence of nested pipelines, which was partially worked around with operator modifiers.
- Context-dependent grammar for operators that was unintuitive for users, and close to impossible to parse and syntax-highlight.
- Inability to define constants with
let, and also soon types, functions, and custom operators within the pipeline definition directly. - Lack of control flow within a pipeline like
if { … } else { … }made it unnecessarily hard to work with your data.
Can I mix-and-match TQL1 and TQL2?
Yes! The legacy operator takes a string as an argument that is parsed as a
TQL1 pipeline and then replaces itself with it. For example:
// tql2
export
where @name == "zeek.notice"
// The `top` operator is not yet available in TQL2, so we fall back to TQL1.
legacy "top msg"
Where can I find the TQL2 reference?
Hang tight! We're in the process of writing a TQL2 reference, and aim to make it
available soon. Until then, you can find all available operators and functions
in TQL2 via show plugins:
show plugins
| unroll types
| put name, type=types
| where "tql2" in type
When should I upgrade?
We do not recommend upgrading production deployments yet, but encourage playing with TQL2 to get a feel for it and to report any missing things that you may need.
Where can I give feedback about TQL2?
Head over to the #developers channel in our community Discord.
What features does TQL2 currently support?
| Status | Explanation |
|---|---|
| ✅ | (mostly) complete |
| 🔧 | in progress |
| Feature | Example | Status | Comments |
|---|---|---|---|
| Simple assignment | conn.test = "done" | ✅ | Insert-or-replace semantics |
| Statement separation | newline or | | ✅ | |
| Dynamic field names | object[expr] | 🔧 | |
| Arbitrary field names | this["@!?$#"] | ✅ | |
| Conditionals | if foo == 42 {...} else {...} | ✅ | |
| Pattern matching | match proto { "UDP" => ..., "TCP" => ... } | 🔧 | |
| High performance | ✅ | Columnar computation engine | |
| Object expression | { foo: 1, bar.baz.qux: 2 } | ✅ | Field names can also be strings |
| List expression | [1, 2, null, 4] | ✅ | |
| Functions/methods | foo(bar) && baz.starts_with("bar") | ✅ | |
| Named arguments | foo.bar(baz=1, qux=2) | ✅ | |
| Arithmetic operators | a + b / c < d - e | ✅ | Also works with durations, etc. |
| Event metadata | where @name == "something" | ✅ | |
| Constant bindings | let $bad_ip = 1.2.3.4 | search $bad_ip | ✅ | |
| Domain types | src in 255.0.0.0/24 && duration > 2min | ✅ | ip, subnet, duration, time, blob |
| Nested pipelines | group protocol {...}, every 1h {...} | 🔧 | |
| Spread operator | {...foo, bar: [42, ...baz, qux]} | ✅ | Can expand objects and lists |
| Packages | my_package::my_function(foo) | 🔧 | |
| OCSF functionality | ocsf::validate, ocsf::class_uid | 🔧 | |
| Type definitions | type flow = {src: ip, dst: ip} | 🔧 | |
| Escape hatches | python "..." | js "..." | 🔧 | |
| Secrets | secret("MY_SECRET") | 🔧 | |
| Function definitions | fn my_function(...) { ... } | 🔧 | |
| Multi-stage parsing | msg.parse_json().content.parse_cef() | ✅ |