Next

Unreleased changes.

Features

Grouped enumeration

The enumerate operator now supports a group option to enumerate events separately based on a value.

For example, to have a field act as a counter for a value, use the following pipeline:

from {x: 1}, {x: 2}, {x: "1"}, {x: 2}
enumerate count, group=x
count = count + 1

{
  count: 1,
  x: 1,
}
{
  count: 1,
  x: 2,
}
{
  count: 1,
  x: "1",
}
{
  count: 2,
  x: 2,
}

By @raxyte in #5475.

Flag for preventing automatic pipeline starts

When the node starts, pipelines that were previously running are immediately started. The new --no-autostart flag can be used to disable this behavior.

By @jachris in #5470.

Lambdas in map and where can capture surrounding fields

Lambda expressions in the map and where functions can now capture and access fields from the surrounding context, enabling more powerful data transformations.

For example:

from {
  host: "server1",
  ports: [80, 443, 8080]
}
ports = ports.map(p => {host: host, port: p})

{
  host: "server1",
  ports: [
    {
      host: "server1",
      port: 80,
    },
    {
      host: "server1",
      port: 443,
    },
    {
      host: "server1",
      port: 8080,
    },
  ],
}

By @raxyte in #5457.

Octet Counting in read_syslog

We have added a new option octet_counting to the read_syslog operator. Enabling this option will determine messages boundaries according to RFC6587 instead of our heuristic.

By @IyeOnline in #5472.

Changes

Dedicated Syslog Schema Names

The read_syslog operator now produces dedicated schemas syslog.rfc5425, syslog.rfc3164 and syslog.unknown instead of an unspecific tenzir.syslog.

By @IyeOnline in #5472.

Keep zeek TSV logs as-is in read_zeek_tsv

Parsing Zeek TSV logs no longer attempts to cast the parsed events to a shipped Zeek schema.

By @tobim in #5461.

Bug Fixes

Improved Syslog Output Schema

We have improved our read_syslog operator and parse_syslog function. They no longer re-order fields if the syslog format changes mid-stream and produce correctly typed null values for the special - value.

By @IyeOnline in #5472.