🚀 Features
Section titled “🚀 Features”PRs 4133-4138-satta
Section titled “PRs 4133-4138-satta”The suricata parser’s schema now more accurately reflects Suricata’s Eve JSON
output, adding many fields that were previously missing.
Add show schemas to display all available schemas
Section titled “Add show schemas to display all available schemas”Apr 19, 2024 · @dominiklohmann · #4131
The show schemas operator lists all unique schemas of events stored at the
node.
Add configurable contexts
Section titled “Add configurable contexts”You can now define contexts and their creation parameters in the
tenzir.contexts section of the configuration file.
Add a delay to retrying failed pipelines
Section titled “Add a delay to retrying failed pipelines”Apr 17, 2024 · @Dakostu · #4108
Stopping a failed pipeline now moves it into the stopped state in the app and
through the /pipeline/update API, stopping automatic restarts on failure.
Pipelines now restart on failure at most every minute. The new API parameter
retry_delay is available in the /pipeline/create, /pipeline/launch, and
/pipeline/update APIs to customize this value. For configured pipelines, the
new restart-on-error option supersedes the previous autostart.failed option
and may be set either to a boolean or to a duration, with the former using the
default retry delay and the latter using a custom one.
The output of show pipelines and the /pipeline/list API now includes the
start time of the pipeline in the field start_time, the newly added retry
delay in the field retry_delay, and whether the pipeline is hidden from the
overview page on app.tenzir.com in the field hidden.
Add value grouping to chart and remove --title
Section titled “Add value grouping to chart and remove --title”Apr 15, 2024 · @jachris · #4119
Some charts supported by the chart operator (bar, line, and area)
now have a --position argument, with the possible values of
grouped and stacked.
Support 0mq inproc sockets
Section titled “Support 0mq inproc sockets”The 0mq connector now supports inproc socket endpoint URLs, allowing you to
create arbitrary publish/subscribe topologies within a node. For example, save zmq inproc://foo writes messages to the in-process socket named foo.
Generalize every to work with all operators
Section titled “Generalize every to work with all operators”Apr 12, 2024 · @dominiklohmann · #4109
The every <duration> operator modifier now supports all operators, turning
blocking operators like tail, sort or summarize into operators that emit
events every <duration>.
Add multi-line syslog message support
Section titled “Add multi-line syslog message support”Apr 8, 2024 · @eliaskosunen · #4080
Syslog messages spanning multiple lines are now supported.
Add a --timeout <duration> option to batch
Section titled “Add a --timeout <duration> option to batch”Apr 5, 2024 · @dominiklohmann · #4095
The batch operator gained a new --timeout <duration> option that controls
the maixmum latency for withholding events for batching.
Add unroll operator
Section titled “Add unroll operator”Mar 28, 2024 · @jachris · #4078
The unroll operator transforms an event that contains a list into a sequence
of events where each output event contains one of the list elements.
Introduce the BITZ format
Section titled “Introduce the BITZ format”Mar 27, 2024 · @dominiklohmann · #4079
The bitz format resembles Tenzir’s internal wire format. It enables lossless
and quick transfer of events between Tenzir nodes through any connector.
Add UDP connector
Section titled “Add UDP connector”The new udp connector comes with a loader and saver to read bytes from and
write bytes to a UDP socket.
Add deduplicate operator
Section titled “Add deduplicate operator”Mar 27, 2024 · @eliaskosunen · #4068
The deduplicate operator allows removing duplicate events based on
specific fields.
🔧 Changes
Section titled “🔧 Changes”Remove events output from many context operators
Section titled “Remove events output from many context operators”Apr 23, 2024 · @dominiklohmann · #4143
The context create, context reset, context update, and context load
operators no return information about the context. Pipelines ending with these
operators will now be considered closed, and you will be asked to deploy them in
the Explorer. Previously, users commonly added discard after these operators
to force this behavior.
Implement a parquet parser and printer
Section titled “Implement a parquet parser and printer”Apr 22, 2024 · @balavinaithirthan · #4116
The parquet format more efficiently reads and writes Parquet files. The format is streamable for write parquet.
Add value grouping to chart and remove --title
Section titled “Add value grouping to chart and remove --title”Apr 15, 2024 · @jachris · #4119
In the chart operator, unless otherwise specified, every field but the
first one is taken to be a value for the Y-axis, instead of just the second one.
If the value for -x/--name or -y/--value is explicitly specified, the
other one must now be too.
The --title option is removed from chart. Titles can instead be set directly
in the web interface.
Support 0mq inproc sockets
Section titled “Support 0mq inproc sockets”The 0mq connector no longer automatically monitors TCP sockets to wait until
at least one remote peer is present. Explicitly pass --monitor for this
behavior.
Remove metrics from /pipeline/list
Section titled “Remove metrics from /pipeline/list”Apr 12, 2024 · @dominiklohmann · #4114
The show pipelines operator and /pipeline/list endpoint no longer include
pipeline metrics. We recommend using the metrics operator instead, which
offers the same data in a more flexible way.
Implement feather printer and parser
Section titled “Implement feather printer and parser”Apr 10, 2024 · @balavinaithirthan · #4089
The feather format now reads and writes Arrow IPC streams in addition to
Feather files, and no longer requires random access to a file to function,
making the format streamable with both read feather and write feather.
Add multi-line syslog message support
Section titled “Add multi-line syslog message support”Apr 8, 2024 · @eliaskosunen · #4080
Lines of input containing an invalid syslog messages are now assumed to be a continuation of a message on a previous line, if there’s any.
Remove many deprecated things
Section titled “Remove many deprecated things”Apr 7, 2024 · @dominiklohmann · #4103
The tenzir-ctl count <expr> command no longer exists. It has long been
deprecated and superseded by pipelines of the form export | where <expr> | summarize count(.).
The deprecated tenzir-ctl status command and the corresponding /status
endpoint no longer exist. They have been superseded by the show and metrics
operators that provide more detailed insight.
The deprecated tenzir.aging-frequency and tenzir.aging-query options no
longer exist. We recommend using the compaction or disk monitor mechanisms
instead to delete persisted events.
🐞 Bug Fixes
Section titled “🐞 Bug Fixes”Fix HTTP saver Content-Length computation
Section titled “Fix HTTP saver Content-Length computation”The http saver now correctly sets the Content-Length header value for HTTP
POST requests.
Remove events output from many context operators
Section titled “Remove events output from many context operators”Apr 23, 2024 · @dominiklohmann · #4143
The enrich operator sometimes stopped working when it encountered an event for
which the specified fields did not exist. This no longer happens.
Fix verification of large FlatBuffers tables
Section titled “Fix verification of large FlatBuffers tables”Apr 22, 2024 · @dominiklohmann · #4137
Lookup tables with more than 1M entries failed to load after the node was restarted. This no longer happens.
Fix a race condition in /serve
Section titled “Fix a race condition in /serve”Apr 16, 2024 · @dominiklohmann · #4123
We fixed a bug that caused the explorer to sometimes show 504 Gateway Timeout errors for pipelines where the first result took over two seconds to arrive.
Support parallel connections in from tcp
Section titled “Support parallel connections in from tcp”Apr 15, 2024 · @dominiklohmann · #4084
The tcp connector now supports accepting multiple connections in parallel when
used with the from operator, parsing data separately per connection.
Fix unflattening of empty records and null records
Section titled “Fix unflattening of empty records and null records”Apr 9, 2024 · @jachris · #4104
Empty records and null values of record type are now correctly unflattened.
Shut down node a configured pipeline fails to start
Section titled “Shut down node a configured pipeline fails to start”Apr 5, 2024 · @dominiklohmann · #4097
Nodes now shut down with a non-zero exit code when pipelines configured as part
of the tenzir.yaml file fail to start, making such configuration errors easier
to spot.
Remove wrong EXPOSE in Dockerfile
Section titled “Remove wrong EXPOSE in Dockerfile”Apr 5, 2024 · @dominiklohmann · #4099
Tenzir Docker images no longer expose 5158/tcp by default, as this prevented running multiple containers in the same network or in host mode.
Fix shutdown of connected pipelines alongside node
Section titled “Fix shutdown of connected pipelines alongside node”Apr 5, 2024 · @dominiklohmann · #4093
Pipelines run with the tenzir binary that connected to a Tenzir Node did
sometimes not shut down correctly when the node shut down. This now happens
reliably.
Fix HTTP PUT with empty request body
Section titled “Fix HTTP PUT with empty request body”We fixed a bug in the http saver that prevented sending HTTP PUT requests with
an empty request body.
Make python operator not discard fields that start with an underscore
Section titled “Make python operator not discard fields that start with an underscore”Apr 4, 2024 · @jachris · #4085
The python operator no longer discards field that start with an underscore.
Fix python deadlock for empty input
Section titled “Fix python deadlock for empty input”Apr 4, 2024 · @jachris · #4086
The python operator no longer deadlocks when given an empty program.
Fix JSON printer handling of inf and nan
Section titled “Fix JSON printer handling of inf and nan”Apr 3, 2024 · @jachris · #4087
The JSON printer previously printed invalid JSON for inf and nan, which
means that serve could sometimes emit invalid JSON, which is not handled well
by platform/app. Instead, we now emit null.
Fix the schema name in show contexts
Section titled “Fix the schema name in show contexts”Mar 28, 2024 · @dominiklohmann · #4082
The schema name of events returned by show contexts sometimes did not match
the type of the context. This now works reliably.