Download the release on GitHub.
Features
Section titled “Features”Add apply operator
Section titled “Add apply operator”The new apply operator includes pipelines defined in other files.
Add parse operator
Section titled “Add parse operator”The new, experimental parse operator applies a parser to the string stored in
a given field.
Add from/load/to/save <uri/file>
Section titled “Add from/load/to/save <uri/file>”The operators from, to, load, and save support using URLs and file paths
directly as their argument. For example, load https://example.com means
load https https://example.com, and save local-file.json means
save file local-file.json.
By @eliaskosunen in #3608.
Add syslog parser
Section titled “Add syslog parser”The syslog parser allows reading both RFC 5424 and RFC 3164 syslog messages.
By @eliaskosunen in #3645.
Add yield operator
Section titled “Add yield operator”The new yield operator extracts nested records with the ability to unfold
lists.
Introduce an experimental python pipeline operator
Section titled “Introduce an experimental python pipeline operator”The python operator adds the ability to perform arbitrary event to event
transformations with the full power of Python 3.
Implement a TCP loader
Section titled “Implement a TCP loader”We added a new tcp connector that allows reading raw bytes
from TCP or TLS connections.
By @dominiklohmann in #3664.
Add file extension detection to from/to
Section titled “Add file extension detection to from/to”When using from <URL> and to <URL> without specifying the format explicitly
using a read/write argument, the default format is determined by the file
extension for all loaders and savers, if possible. Previously, that was only
done when using the file loader/saver. Additionally, if the file name would
indicate some sort of compression (e.g. .gz), compression and decompression is
performed automatically. For example, from https://example.com/myfile.yml.gz
is expanded to load https://example.com/myfile.yml.gz | decompress gzip | read yaml automatically.
By @eliaskosunen in #3653.
Support show-ing all aspects at once
Section titled “Support show-ing all aspects at once”Use show without an aspect to return information about all aspects of a node.
By @dominiklohmann in #3650.
Implement context backends for the contextualizer
Section titled “Implement context backends for the contextualizer”The closed-source context plugin offers a backend functionality for
finding matches between data sets.
The new lookup-table built-in is a hashtable-based
contextualization algorithm that enriches events based on a unique value.
The JSON format has a new --arrays-of-objects parameter that allows for
parsing a JSON array of JSON objects into an event for each object.
Add ‘min_events’ parameters to /serve endpoint
Section titled “Add ‘min_events’ parameters to /serve endpoint”We optimized the behavior of the ‘serve’ operator to respond
quicker and cause less system load for pipelines that take a
long time to generate the first result. The new min_events
parameter can be used to implement long-polling behavior for
clients of /serve.
Support comments in xsv parser
Section titled “Support comments in xsv parser”Use --allow-comments with the xsv parser (incl. csv, tsv, and ssv)
to treat lines beginning with '#' as comments.
By @eliaskosunen in #3681.
Add export --internal to access metrics
Section titled “Add export --internal to access metrics”The new --internal flag for the export operators returns internal events
collected by the system, for example pipeline metrics.
Changes
Section titled “Changes”Rename pytenzir to tenzir
Section titled “Rename pytenzir to tenzir”We renamed the name of our python package from pytenzir to tenzir.
Implement a TCP loader
Section titled “Implement a TCP loader”We renamed the --bind option of the zmq connector
to --listen.
By @dominiklohmann in #3664.
Consider discard, export, and import as internal operators
Section titled “Consider discard, export, and import as internal operators”Ingress and egress metrics for pipelines now indicate whether the pipeline
sent/received events to/from outside of the node with a new internal flag. For
example, when using the export operator, data is entering the pipeline from
within the node, so its ingress is considered internal.
By @dominiklohmann in #3658.
Bug Fixes
Section titled “Bug Fixes”Add export --internal to access metrics
Section titled “Add export --internal to access metrics”export --live now respects a subsequent where <expr> instead of silently
discarding the filter expression.
Support lists and null values and empty strings in XSV parser
Section titled “Support lists and null values and empty strings in XSV parser”The csv, ssv, and tsv parsers now correctly support empty strings, lists,
and null values.
The tail operator no longer hangs occasionally.
By @dominiklohmann in #3687.
Fix sort type check
Section titled “Fix sort type check”Using the sort operator with polymorphic inputs no longer leads to a failing
assertion under some circumstances.