The operator now supports event-dependent topics, making routing between pipelines more flexible. Additionally, new and operators make taking apart custom logs easier than before.
🚀 Features
Section titled “🚀 Features”Publishing to dynamic topics
Section titled “Publishing to dynamic topics”Jun 24, 2025 · @dominiklohmann · #5294
The publish operator now allows for dynamic topics to be derived from each
individual event.
For example, assuming Suricata logs, publish f"suricata.{event_type}" now
publishes to the topic suricata.alert for alert events and suricata.flow for
flow events. This works with any expression that evaluates to a string,
including publish @name to use the schema name of the event.
HTTP request metadata
Section titled “HTTP request metadata”Jun 24, 2025 · @raxyte · #5295
The from_http operator now supports the metadata_field option when using the
server mode and not just client mode. The request metadata has the following
schema:
| Field | Type | Description |
|---|---|---|
headers | record | The request headers. |
query | record | The query parameters of the request. |
path | string | The path requested. |
fragment | string | The URI fragment of the request. |
method | string | The HTTP method of the request. |
version | string | The HTTP version of the request. |
read_delimited and read_delimited_regex
Section titled “read_delimited and read_delimited_regex”Jun 23, 2025 · @dominiklohmann · #5291
TQL now supports two new operators for parsing data streams with custom
delimiters: read_delimited and read_delimited_regex. These operators provide a more
intuitive and discoverable way to split data on custom separators compared to
the deprecated split_at_regex option in read_lines.
The read_delimited operator splits input on exact string or blob matches:
load_file "data.txt"read_delimited "||"The read_delimited_regex operator splits input using regular expression patterns:
load_tcp "0.0.0.0:514" { read_delimited_regex "(?=<[0-9]+>)"}Both operators support binary data processing and optionally including the
separator in the output. The split_at_regex option in read_lines is now
deprecated in favor of these dedicated operators.
🐞 Bug Fixes
Section titled “🐞 Bug Fixes”OCSF -dev versions
Section titled “OCSF -dev versions”Jun 24, 2025 · @jachris · #5296
The ocsf::apply operator can now be used with the newest development version
of OCSF (v1.6.0-dev). Previously, it claimed that this version does not exist.
Fixed panic in write_parquet
Section titled “Fixed panic in write_parquet”Jun 20, 2025 · @dominiklohmann · #5293
The write_parquet operator no longer panics when specifying
compression_type="snappy" without a compression_level.