Skip to main content
Version: v4.21

TQL2 Migration

This page answers the most frequently asked questions about TQL2.

What is TQL2?

TQL2 is the next generation of the Tenzir Query Language for writing pipelines.

How do I use TQL2?

Start your pipeline with a // tql2 comment to opt into using the new language. This is supported for Tenzir v4.19 or newer.

Why create an all-new language?

TQL1 has grown historically. Over time, we identified the following shortcomings that TQL2 fixes:

  • Lack of composable expressions, in particular functions and arithmetic operations.
  • Absence of nested pipelines, which was partially worked around with operator modifiers.
  • Context-dependent grammar for operators that was unintuitive for users, and close to impossible to parse and syntax-highlight.
  • Inability to define constants with let, and also soon types, functions, and custom operators within the pipeline definition directly.
  • Lack of control flow within a pipeline like if { … } else { … } made it unnecessarily hard to work with your data.

Can I mix-and-match TQL1 and TQL2?

Yes! The legacy operator takes a string as an argument that is parsed as a TQL1 pipeline and then replaces itself with it. For example:

// tql2
export
where @name == "zeek.notice"
// The `top` operator is not yet available in TQL2, so we fall back to TQL1.
legacy "top msg"

Where can I find the TQL2 reference?

Hang tight! We're in the process of writing a TQL2 reference, and aim to make it available soon. Until then, you can find all available operators and functions in TQL2 via show plugins:

show plugins
| unroll types
| put name, type=types
| where "tql2" in type

When should I upgrade?

We do not recommend upgrading production deployments yet, but encourage playing with TQL2 to get a feel for it and to report any missing things that you may need.

Where can I give feedback about TQL2?

Head over to the #developers channel in our community Discord.

What features does TQL2 currently support?

StatusExplanation
(mostly) complete
🔧in progress
FeatureExampleStatusComments
Simple assignmentconn.test = "done"Insert-or-replace semantics
Statement separationnewline or |
Dynamic field namesobject[expr]🔧
Arbitrary field namesthis["@!?$#"]
Conditionalsif foo == 42 {...} else {...}
Pattern matchingmatch proto { "UDP" => ..., "TCP" => ... }🔧
High performanceColumnar computation engine
Object expression{ foo: 1, bar.baz.qux: 2 }Field names can also be strings
List expression[1, 2, null, 4]
Functions/methodsfoo(bar) && baz.starts_with("bar")
Named argumentsfoo.bar(baz=1, qux=2)
Arithmetic operatorsa + b / c < d - eAlso works with durations, etc.
Event metadatawhere @name == "something"
Constant bindingslet $bad_ip = 1.2.3.4 | search $bad_ip
Domain typessrc in 255.0.0.0/24 && duration > 2minip, subnet, duration, time, blob
Nested pipelinesgroup protocol {...}, every 1h {...}🔧
Spread operator{...foo, bar: [42, ...baz, qux]}Can expand objects and lists
Packagesmy_package::my_function(foo)🔧
OCSF functionalityocsf::validate, ocsf::class_uid🔧
Type definitionstype flow = {src: ip, dst: ip}🔧
Escape hatchespython "..." | js "..."🔧
Secretssecret("MY_SECRET")🔧
Function definitionsfn my_function(...) { ... }🔧
Multi-stage parsingmsg.parse_json().content.parse_cef()