🚀 Features
Section titled “🚀 Features”Add export --internal to access metrics
Section titled “Add export --internal to access metrics”Dec 1, 2023 · @jachris · #3619
The new --internal flag for the export operators returns internal events
collected by the system, for example pipeline metrics.
Add parse operator
Section titled “Add parse operator”Dec 1, 2023 · @jachris · #3665
The new, experimental parse operator applies a parser to the string stored in
a given field.
Add apply operator
Section titled “Add apply operator”Dec 1, 2023 · @jachris · #3677
The new apply operator includes pipelines defined in other files.
Implement context backends for the contextualizer
Section titled “Implement context backends for the contextualizer”Dec 1, 2023 · @Dakostu · #3684
The closed-source context plugin offers a backend functionality for
finding matches between data sets.
The new lookup-table built-in is a hashtable-based
contextualization algorithm that enriches events based on a unique value.
The JSON format has a new --arrays-of-objects parameter that allows for
parsing a JSON array of JSON objects into an event for each object.
Implement a TCP loader
Section titled “Implement a TCP loader”Dec 1, 2023 · @dominiklohmann · #3664
We added a new tcp connector that allows reading raw bytes
from TCP or TLS connections.
Support comments in xsv parser
Section titled “Support comments in xsv parser”Nov 29, 2023 · @eliaskosunen · #3681
Use --allow-comments with the xsv parser (incl. csv, tsv, and ssv)
to treat lines beginning with '#' as comments.
Add file extension detection to from/to
Section titled “Add file extension detection to from/to”Nov 27, 2023 · @eliaskosunen · #3653
When using from <URL> and to <URL> without specifying the format explicitly
using a read/write argument, the default format is determined by the file
extension for all loaders and savers, if possible. Previously, that was only
done when using the file loader/saver. Additionally, if the file name would
indicate some sort of compression (e.g. .gz), compression and decompression is
performed automatically. For example, from https://example.com/myfile.yml.gz
is expanded to load https://example.com/myfile.yml.gz | decompress gzip | read yaml automatically.
Support show-ing all aspects at once
Section titled “Support show-ing all aspects at once”Nov 26, 2023 · @dominiklohmann · #3650
Use show without an aspect to return information about all aspects of a node.
Add ‘min_events’ parameters to /serve endpoint
Section titled “Add ‘min_events’ parameters to /serve endpoint”We optimized the behavior of the ‘serve’ operator to respond
quicker and cause less system load for pipelines that take a
long time to generate the first result. The new min_events
parameter can be used to implement long-polling behavior for
clients of /serve.
Add syslog parser
Section titled “Add syslog parser”Nov 22, 2023 · @eliaskosunen · #3645
The syslog parser allows reading both RFC 5424 and RFC 3164 syslog messages.
Add yield operator
Section titled “Add yield operator”Nov 21, 2023 · @jachris · #3651
The new yield operator extracts nested records with the ability to unfold
lists.
Introduce an experimental python pipeline operator
Section titled “Introduce an experimental python pipeline operator”The python operator adds the ability to perform arbitrary event to event
transformations with the full power of Python 3.
Add from/load/to/save <uri/file>
Section titled “Add from/load/to/save <uri/file>”Nov 17, 2023 · @eliaskosunen · #3608
The operators from, to, load, and save support using URLs and file paths
directly as their argument. For example, load https://example.com means
load https https://example.com, and save local-file.json means
save file local-file.json.
🔧 Changes
Section titled “🔧 Changes”Implement a TCP loader
Section titled “Implement a TCP loader”Dec 1, 2023 · @dominiklohmann · #3664
We renamed the --bind option of the zmq connector
to --listen.
Rename pytenzir to tenzir
Section titled “Rename pytenzir to tenzir”We renamed the name of our python package from pytenzir to tenzir.
Consider discard, export, and import as internal operators
Section titled “Consider discard, export, and import as internal operators”Nov 21, 2023 · @dominiklohmann · #3658
Ingress and egress metrics for pipelines now indicate whether the pipeline
sent/received events to/from outside of the node with a new internal flag. For
example, when using the export operator, data is entering the pipeline from
within the node, so its ingress is considered internal.
🐞 Bug Fixes
Section titled “🐞 Bug Fixes”Add export --internal to access metrics
Section titled “Add export --internal to access metrics”Dec 1, 2023 · @jachris · #3619
export --live now respects a subsequent where <expr> instead of silently
discarding the filter expression.
Support lists and null values and empty strings in XSV parser
Section titled “Support lists and null values and empty strings in XSV parser”Nov 30, 2023 · @dominiklohmann · #3687
The csv, ssv, and tsv parsers now correctly support empty strings, lists,
and null values.
The tail operator no longer hangs occasionally.
Fix sort type check
Section titled “Fix sort type check”Nov 20, 2023 · @jachris · #3655
Using the sort operator with polymorphic inputs no longer leads to a failing
assertion under some circumstances.