Download the release on GitHub.
Features
Section titled “Features”Support concepts in more places
Section titled “Support concepts in more places”Concepts are now supported in more places than just the where operator: All
operators and concepts that reference fields in events now support them
transparently. For example, it is not possible to enrich with a lookup table
against all source IP addresses defined in the concept net.src.ip, or to group
by destination ports across different schemas with the concept net.dst.port.
By @dominiklohmann in #3812.
Implement timeshift and delay operators
Section titled “Implement timeshift and delay operators”The new timeshift operator adjusts timestamps relative to a given start time,
with an optional speedup.
The new delay operator delays events relative to a given start time, with an
optional speedup.
By @dominiklohmann in #3701.
Add GELF parser and document Graylog integration
Section titled “Add GELF parser and document Graylog integration”The new gelf parser reads a stream of NULL-byte terminated messages in Graylog
Extended Log Format (GELF).
Update the main repository to include the pipeline manager autostart changes
Section titled “Update the main repository to include the pipeline manager autostart changes”Pipeline states in the /pipeline API will not change upon node shutdown
anymore. When a node restarts afterwards, previously running pipelines will
continue to run while paused pipelines will load in a stopped state.
Add node health metrics
Section titled “Add node health metrics”A Tenzir node will now automatically collect and store metrics about disk, cpu and memory usage of the host machine.
Add xsv --no-header
Section titled “Add xsv --no-header”The csv, tsv, ssv and xsv printers now support not printing a header
line with the --no-header option.
By @eliaskosunen in #3821.
Add savers for curl connectors
Section titled “Add savers for curl connectors”The http and https loaders now also have savers to send data from a pipeline
to a remote API.
The http and https connectors have a new flag --form to submit the request
body URL-encoded. This also changes the Content-Type header to
application/x-www-form-urlencoded.
Allow setting the header for csv, tsv, and ssv manually
Section titled “Allow setting the header for csv, tsv, and ssv manually”The csv, tsv, ssv and xsv parsers now support setting the header line
manually with the --header option.
By @dominiklohmann in #3778.
Implement the diagnostics operator
Section titled “Implement the diagnostics operator”The new diagnostics operator provides information about diagnostics that a
pipeline may encounter during its lifetime.
Fix export --live and introduce metrics
Section titled “Fix export --live and introduce metrics”The metrics operator returns internal metrics events generated in a Tenzir
node. Use metrics --live to get a feed of metrics as they are being generated.
By @dominiklohmann in #3790.
Add support for Cisco Firepower syslog messages
Section titled “Add support for Cisco Firepower syslog messages”The RFC 3164 syslog parser now supports years in the message timestamp.
By @eliaskosunen in #3833.
Improve handling of open file descriptors
Section titled “Improve handling of open file descriptors”On Linux systems, the process metrics now have an additional
value open_fds showing the number of file descriptors
opened by the node.
Implement the lookup operator
Section titled “Implement the lookup operator”The new lookup operator performs live filtering of the import feed using a
context, and translates context updates into historical queries. This
effectively enables live and retro matching in a single operator.
Add time parser
Section titled “Add time parser”The time parser allows parsing datetimes and timestamps from arbitrary
strings using a strptime-like format string.
By @eliaskosunen in #3738.
Changes
Section titled “Changes”Switch from JSON to MsgPack data transport
Section titled “Switch from JSON to MsgPack data transport”The fluent-bit source operator no longer performs JSON conversion from
Fluent Bit prior to processing an event. Instead, it directly processes the
MsgPack data that Fluent Bit uses internally for more robust and quicker event
delivery.
Bug Fixes
Section titled “Bug Fixes”Print Zeek TSV metadata when schema changes
Section titled “Print Zeek TSV metadata when schema changes”The zeek-tsv printer incorrectly emitted metadata too frequently. It now only
writes opening and closing tags when it encounters a new schema.
By @dominiklohmann in #3836.
Optimize pipeline when using /pipeline/launch
Section titled “Optimize pipeline when using /pipeline/launch”The /pipeline/launch endpoint now optimizes the pipeline before starting it.
Fail properly when transfer breaks
Section titled “Fail properly when transfer breaks”Failing transfers using http(s) and ftp(s) connectors now properly return an
error when the transfer broke. For example, from http://does.not.exist no
longer returns silently a success.
Fix blob parsing with padding
Section titled “Fix blob parsing with padding”When reading Base64-encoded JSON strings with the blob type, = padding is
now accepted.
Fix the tenzir/tenzir:latest-slim image
Section titled “Fix the tenzir/tenzir:latest-slim image”The tenzir/tenzir:latest-slim Docker image now sets a default
TENZIR_CACHE_DIRECTORY automatically.
By @dominiklohmann in #3764.
Display failing pipeline diagnostics in /serve
Section titled “Display failing pipeline diagnostics in /serve”The /serve API now displays why a pipeline became unavailable in an error case
instead of showing a generic error message. This causes runtime errors in
pipelines to show up in the Explorer on app.tenzir.com.
By @dominiklohmann in #3788.
Fix export --live and introduce metrics
Section titled “Fix export --live and introduce metrics”export --live sometimes got stuck, failing to deliver events. This no longer
happens.
By @dominiklohmann in #3790.
Override lookup-table context entries for duplicate keys
Section titled “Override lookup-table context entries for duplicate keys”Updating entries of a lookup-table context now overrides values with duplicate
keys instead of ignoring them.
By @dominiklohmann in #3808.