🚀 Features
Section titled “🚀 Features”Implement a context content dumping mechanism
Section titled “Implement a context content dumping mechanism”Feb 21, 2024 · @Dakostu · #3893
The new context inspect <context-name> command dumps a specific context’s
user-provided data, usually the context’s content.
Support parsing numeric timestamps since epoch
Section titled “Support parsing numeric timestamps since epoch”Feb 20, 2024 · @jachris · #3927
When specifying a schema with a field typed as time #unit=<unit>, numeric
values will be interpreted as offsets from the epoch.
Implement more malleable lookup data for contexts
Section titled “Implement more malleable lookup data for contexts”Feb 19, 2024 · @Dakostu · #3920
The context match events now contain a new field mode that states the lookup
mode of this particular match.
The enrich operator gained a --filter option, which causes it to exclude
enriched events that do not contain a context.
Add running and paused times to pipeline metrics
Section titled “Add running and paused times to pipeline metrics”Feb 16, 2024 · @dominiklohmann · #3940
Operator metrics now separately track the time that an operator was paused or
running in the time_paused and time_running values in addition to the
wall-clock time in time_total. Throughput rates now exclude the paused time
from their calculation.
Implement context load, context save, and context reset
Section titled “Implement context load, context save, and context reset”Feb 13, 2024 · @eliaskosunen · #3908
The context reset operator allows for clearing the state of a context.
The context save and context load operators allow serializing and
deserializing the state of a context to/from bytes.
Add new ‘—file’ option to the python operator
Section titled “Add new ‘—file’ option to the python operator”The python operator gained a new --file flag
that allows loading python code from a file instead
of providing it as part of the pipeline definition.
Handle nested fields and integers as selectors in JSON parser
Section titled “Handle nested fields and integers as selectors in JSON parser”Feb 5, 2024 · @jachris · #3900
The --selector option of the json parser now works with nested fields, and
integer fields.
Improve the export operator
Section titled “Improve the export operator”Feb 5, 2024 · @dominiklohmann · #3909
The export operator gained a --low-priority option, which causes it to
interfere less with regular priority exports at the cost of potentially running
slower.
Update the main repository to include the pipeline run ID
Section titled “Update the main repository to include the pipeline run ID”Feb 1, 2024 · @Dakostu · #3883
Managed pipelines now contain a new total_runs parameter that counts all
started runs. The new run field is available in the events delivered by the
metrics and diagnostics operators.
Add the openapi operator
Section titled “Add the openapi operator”Feb 1, 2024 · @dominiklohmann · #3898
The openapi source operator generates Tenzir’s OpenAPI specification. Use
openapi | to ./openapi.yaml to generate a file with the canonical format.
Add structured_data to syslog output
Section titled “Add structured_data to syslog output”Jan 31, 2024 · @eliaskosunen · #3871
The structured_data field in RFC 5424-style syslog messages is
now parsed and included in the output.
Add lines printer
Section titled “Add lines printer”Jan 31, 2024 · @eliaskosunen · #3847
The lines printer enables simple line-delimited formatting of events.
Add Bloom filter context
Section titled “Add Bloom filter context”The new bloom-filter context represents large sets in a space-efficient
manner.
Update the main repository to include timestamped pipelines
Section titled “Update the main repository to include timestamped pipelines”Jan 30, 2024 · @Dakostu · #3869
show pipelines and the /pipeline API endpoints now include created_at and last_modified fields that track the pipeline’s creation and last manual modification time, respectively. Pipelines created with older versions of Tenzir will use the start time of the node as their creation time.
Rewrite chart and set-attributes operators
Section titled “Rewrite chart and set-attributes operators”Jan 30, 2024 · @eliaskosunen · #3866
The chart operator adds metadata to the schema of the input events,
enabling rendering events as bar, area, line, or pie charts on app.tenzir.com.
🔧 Changes
Section titled “🔧 Changes”Context versioning
Section titled “Context versioning”Feb 21, 2024 · @eliaskosunen · #3945
The binary format used by contexts for saving on disk on node shutdown is now versioned. A node can support loading of multiple different versions, and automigrate between them.
Disable colors if NO_COLOR or not a terminal
Section titled “Disable colors if NO_COLOR or not a terminal”Feb 19, 2024 · @jachris · #3952
Color escape codes are no longer emitted if NO_COLOR is set to a non-empty
value, or when the output device is not a terminal.
Remove reader, writer, and language plugin types
Section titled “Remove reader, writer, and language plugin types”Feb 1, 2024 · @dominiklohmann · #3899
We removed the tenzir-ctl start subcommand. Users should switch to
the tenzir-node command instead, which accepts the same arguments
and presents the same command-line interface.
Allow plugins to bundle further builtins
Section titled “Allow plugins to bundle further builtins”Jan 31, 2024 · @dominiklohmann · #3877
Plugins may now depend on other plugins. Plugins with unmet dependencies are
automatically disabled. For example, the lookup and enrich plugins now
depend on the context plugin. Run show plugins to see all available plugins
and their dependencies.
Replace tenzir.db-directory with tenzir.state-directory
Section titled “Replace tenzir.db-directory with tenzir.state-directory”Jan 31, 2024 · @dominiklohmann · #3889
The option tenzir.db-directory is deprecated in favor of the
tenzir.state-directory option and will be removed in the future.
🐞 Bug Fixes
Section titled “🐞 Bug Fixes”Fix restart on failure
Section titled “Fix restart on failure”Feb 19, 2024 · @dominiklohmann · #3947
The option to automatically restart on failure did not correctly trigger for pipelines that failed an operator emitted an error diagnostic, a new mechanism for improved error messages introduced with Tenzir v4.8. Such pipelines now restart automatically as expected.
Prevent duplicate fields in schema
Section titled “Prevent duplicate fields in schema”Feb 12, 2024 · @jachris · #3929
Invalid schema definitions, where a record contains the same key multiple times, are now detected and rejected.
Add support for commas in seconds in the time data parser
Section titled “Add support for commas in seconds in the time data parser”Feb 8, 2024 · @eliaskosunen · #3903
Commas are now allowed as subsecond separators in timestamps in TQL. Previously, only dots were allowed, but ISO 8601 allows for both.
Fix logger deadlock in python tests
Section titled “Fix logger deadlock in python tests”We fixed a rare deadlock by changing the internal logger behavior from blocking until the oldest messages were consumed to overwriting them.
Update the repository to include lookup lifetime fixes
Section titled “Update the repository to include lookup lifetime fixes”Feb 6, 2024 · @Dakostu · #3910
Retroactive lookups will now properly terminate when they have finished.
Improve the export operator
Section titled “Improve the export operator”Feb 5, 2024 · @dominiklohmann · #3909
We fixed a bug that under rare circumstances led to an indefinite hang when using a high-volume source followed by a slow transformation and a fast sink.
Gracefully handle misaligned header and values in xsv parser
Section titled “Gracefully handle misaligned header and values in xsv parser”Feb 4, 2024 · @dominiklohmann · #3874
The xsv parser (and by extension the csv, tsv, and ssv parsers) skipped
lines that had a mismatch between the number of values contained and the number
of fields defined in the header. Instead, it now fills in null values for
missing values and, if the new --auto-expand option is set, also adds new
header fields for excess values.
Make /serve more consistent
Section titled “Make /serve more consistent”Jan 31, 2024 · @dominiklohmann · #3885
The /serve API sometimes returned an empty string for the next continuation
token instead of null when there are no further results to fetch. It now
consistently returns null.