Lambda Captures

This release enhances TQL’s data transformation capabilities with lambda expressions that can capture surrounding fields in map and where functions, plus grouped enumeration for separate event counting. We’ve also improved operator composability with enhanced to_splunk parameters, added octet counting support for syslog messages, and fixed critical issues in Kafka message handling and HTTP request processing.

Download the release on GitHub.

Features

Grouped enumeration

The enumerate operator now supports a group option to enumerate events separately based on a value.

For example, to have a field act as a counter for a value, use the following pipeline:

from {x: 1}, {x: 2}, {x: "1"}, {x: 2}
enumerate count, group=x
count = count + 1

{
  count: 1,
  x: 1,
}
{
  count: 1,
  x: 2,
}
{
  count: 1,
  x: "1",
}
{
  count: 2,
  x: 2,
}

By @raxyte in #5475.

Flag for preventing automatic pipeline starts

When the node starts, pipelines that were previously running are immediately started. The new --no-autostart flag can be used to disable this behavior.

By @jachris in #5470.

Lambdas in map and where can capture surrounding fields

Lambda expressions in the map and where functions can now capture and access fields from the surrounding context, enabling more powerful data transformations.

For example:

from {
  host: "server1",
  ports: [80, 443, 8080]
}
ports = ports.map(p => {host: host, port: p})

{
  host: "server1",
  ports: [
    {
      host: "server1",
      port: 80,
    },
    {
      host: "server1",
      port: 443,
    },
    {
      host: "server1",
      port: 8080,
    },
  ],
}

By @raxyte in #5457.

Improve to_splunk composability

We have improved the composability of the to_splunk operator. The host and source parameters now accept a string-expression instead of only a constant. Further, there is a new event parameter that can be used to specify what should be send as the event to the Splunk HTTP Event Collector.

The combination of these options improves the composability of the operator, allowing you to set event-specific Splunk parameters, while not also transmitting them as part of the actual event:

from {
  host: "my-host",
  a: 42,
  b: 0
}

// move the entire event into `event`
this = { event: this }

// hoist the splunk specific field back out
move host = event.host

to_splunk "https://localhost:8088",
  hec_token=secret("splunk-hec-token"),
  host=host,
  event=event

By @IyeOnline in #5478.

Octet Counting in read_syslog

We have added a new option octet_counting to the read_syslog operator. Enabling this option will determine messages boundaries according to RFC6587 instead of our heuristic.

By @IyeOnline in #5472.

Changes

Dedicated Syslog Schema Names

The read_syslog operator now produces dedicated schemas syslog.rfc5425, syslog.rfc3164 and syslog.unknown instead of an unspecific tenzir.syslog.

By @IyeOnline in #5472.

Keep zeek TSV logs as-is in read_zeek_tsv

Parsing Zeek TSV logs no longer attempts to cast the parsed events to a shipped Zeek schema.

By @tobim in #5461.

Bug Fixes

Explicit Commits in load_kafka

The load_kafka operator now explicitly commits messages it has consumed. By default, it will commit every 1000 messages or every 10 seconds, with the behavior being customizable via two new operator arguments.

Previously, the operator would commit every message asynchronously loaded by the backing library automatically, which may have included messages that were never accepted by the pipeline.

By @IyeOnline in #5465.

http operator stalling

The http operator now correctly handles its internal waiting state, fixing an intermittent issue where HTTP requests could hang unexpectedly.

By @raxyte in #5479.

Improved Syslog Output Schema

We have improved our read_syslog operator and parse_syslog function. They no longer re-order fields if the syslog format changes mid-stream and produce correctly typed null values for the special - value.

By @IyeOnline in #5472.

Fixed to_kafka crash

The recently released to_kafka operator would fail with an internal error when used without specifying the message argument.

The operator now works as expected, sending the entire event if the argument is not specified.

By @IyeOnline in #5465.