🚀 Features
Section titled “🚀 Features”Implement the diagnostics operator
Section titled “Implement the diagnostics operator”Jan 22, 2024 · @Dakostu · #3828
The new diagnostics operator provides information about diagnostics that a
pipeline may encounter during its lifetime.
Add support for Cisco Firepower syslog messages
Section titled “Add support for Cisco Firepower syslog messages”Jan 22, 2024 · @eliaskosunen · #3833
The RFC 3164 syslog parser now supports years in the message timestamp.
Add xsv --no-header
Section titled “Add xsv --no-header”Jan 19, 2024 · @eliaskosunen · #3821
The csv, tsv, ssv and xsv printers now support not printing a header
line with the --no-header option.
Add savers for curl connectors
Section titled “Add savers for curl connectors”The http and https loaders now also have savers to send data from a pipeline
to a remote API.
The http and https connectors have a new flag --form to submit the request
body URL-encoded. This also changes the Content-Type header to
application/x-www-form-urlencoded.
Improve handling of open file descriptors
Section titled “Improve handling of open file descriptors”On Linux systems, the process metrics now have an additional
value open_fds showing the number of file descriptors
opened by the node.
Add node health metrics
Section titled “Add node health metrics”A Tenzir node will now automatically collect and store metrics about disk, cpu and memory usage of the host machine.
Support concepts in more places
Section titled “Support concepts in more places”Jan 12, 2024 · @dominiklohmann · #3812
Concepts are now supported in more places than just the where operator: All
operators and concepts that reference fields in events now support them
transparently. For example, it is not possible to enrich with a lookup table
against all source IP addresses defined in the concept net.src.ip, or to group
by destination ports across different schemas with the concept net.dst.port.
Add time parser
Section titled “Add time parser”Jan 12, 2024 · @eliaskosunen · #3738
The time parser allows parsing datetimes and timestamps from arbitrary
strings using a strptime-like format string.
Fix export --live and introduce metrics
Section titled “Fix export --live and introduce metrics”Jan 11, 2024 · @dominiklohmann · #3790
The metrics operator returns internal metrics events generated in a Tenzir
node. Use metrics --live to get a feed of metrics as they are being generated.
Update the main repository to include the pipeline manager autostart changes
Section titled “Update the main repository to include the pipeline manager autostart changes”Jan 11, 2024 · @Dakostu · #3785
Pipeline states in the /pipeline API will not change upon node shutdown
anymore. When a node restarts afterwards, previously running pipelines will
continue to run while paused pipelines will load in a stopped state.
Allow setting the header for csv, tsv, and ssv manually
Section titled “Allow setting the header for csv, tsv, and ssv manually”Jan 9, 2024 · @dominiklohmann · #3778
The csv, tsv, ssv and xsv parsers now support setting the header line
manually with the --header option.
Add GELF parser and document Graylog integration
Section titled “Add GELF parser and document Graylog integration”The new gelf parser reads a stream of NULL-byte terminated messages in Graylog
Extended Log Format (GELF).
Implement timeshift and delay operators
Section titled “Implement timeshift and delay operators”Dec 23, 2023 · @dominiklohmann · #3701
The new timeshift operator adjusts timestamps relative to a given start time,
with an optional speedup.
The new delay operator delays events relative to a given start time, with an
optional speedup.
Implement the lookup operator
Section titled “Implement the lookup operator”Dec 22, 2023 · @Dakostu · #3721
The new lookup operator performs live filtering of the import feed using a
context, and translates context updates into historical queries. This
effectively enables live and retro matching in a single operator.
🔧 Changes
Section titled “🔧 Changes”Switch from JSON to MsgPack data transport
Section titled “Switch from JSON to MsgPack data transport”The fluent-bit source operator no longer performs JSON conversion from
Fluent Bit prior to processing an event. Instead, it directly processes the
MsgPack data that Fluent Bit uses internally for more robust and quicker event
delivery.
🐞 Bug Fixes
Section titled “🐞 Bug Fixes”Print Zeek TSV metadata when schema changes
Section titled “Print Zeek TSV metadata when schema changes”Jan 21, 2024 · @dominiklohmann · #3836
The zeek-tsv printer incorrectly emitted metadata too frequently. It now only
writes opening and closing tags when it encounters a new schema.
Fail properly when transfer breaks
Section titled “Fail properly when transfer breaks”Failing transfers using http(s) and ftp(s) connectors now properly return an
error when the transfer broke. For example, from http://does.not.exist no
longer returns silently a success.
Display failing pipeline diagnostics in /serve
Section titled “Display failing pipeline diagnostics in /serve”Jan 18, 2024 · @dominiklohmann · #3788
The /serve API now displays why a pipeline became unavailable in an error case
instead of showing a generic error message. This causes runtime errors in
pipelines to show up in the Explorer on app.tenzir.com.
Override lookup-table context entries for duplicate keys
Section titled “Override lookup-table context entries for duplicate keys”Jan 12, 2024 · @dominiklohmann · #3808
Updating entries of a lookup-table context now overrides values with duplicate
keys instead of ignoring them.
Fix export --live and introduce metrics
Section titled “Fix export --live and introduce metrics”Jan 11, 2024 · @dominiklohmann · #3790
export --live sometimes got stuck, failing to deliver events. This no longer
happens.
Optimize pipeline when using /pipeline/launch
Section titled “Optimize pipeline when using /pipeline/launch”Jan 11, 2024 · @jachris · #3801
The /pipeline/launch endpoint now optimizes the pipeline before starting it.
Fix blob parsing with padding
Section titled “Fix blob parsing with padding”Jan 11, 2024 · @jachris · #3765
When reading Base64-encoded JSON strings with the blob type, = padding is
now accepted.
Fix the tenzir/tenzir:latest-slim image
Section titled “Fix the tenzir/tenzir:latest-slim image”Dec 21, 2023 · @dominiklohmann · #3764
The tenzir/tenzir:latest-slim Docker image now sets a default
TENZIR_CACHE_DIRECTORY automatically.