Download the release on GitHub.
Features
Section titled “Features”Implement encode_base64 and decode_base64
Section titled “Implement encode_base64 and decode_base64”The new functions encode_base64 and decode_base64 encode and
decode blobs and strings as Base64.
Implement append, prepend, and concatenate
Section titled “Implement append, prepend, and concatenate”The new append, prepend, and concatenate functions add an element to the
end of a list, to the front of a list, and merge two lists, respectively.
xs.append(y) is equivalent to [...xs, y], xs.prepend(y) is equivalent to
[y, ...xs], and concatenate(xs, ys) is equivalent to [...xs, ..ys].
By @dominiklohmann in #4792.
Implement otherwise(<expr>, <expr>)
Section titled “Implement otherwise(<expr>, <expr>)”The function otherwise(primary:any, fallback:any) provides a simple way
to specify a fallback expression when the primary expression
evaluates to null.
Port unroll to TQL2
Section titled “Port unroll to TQL2”The unroll operator is now available in TQL2. It takes a field of type list,
and duplicates the surrounding event for every element of the list.
By @dominiklohmann in #4736.
Support decapsulating SLL2 packets
Section titled “Support decapsulating SLL2 packets”The decapsulate function now handles SLL2 frames (Linux cooked capture
encapsulation).
Implement where and map on lists
Section titled “Implement where and map on lists”The <list>.map(<capture>, <expression>) function replaces each value from
<list> with the value from <expression>. Within <expression>, the elements
are available as <capture>. For example, to add 5 to all elements in the list
xs, use xs = xs.map(x, x + 5).
The <list>.where(<capture>, <predicate>) removes all elements from <list>
for which the <predicate> evaluates to false. Within <predicate>, the
elements are available as <capture>. For example, to remove all elements
smaller than 3 from the list xs, use xs = xs.where(x, x >= 3).
By @dominiklohmann in #4788.
Implement encode_hex() and decode_hex()
Section titled “Implement encode_hex() and decode_hex()”The functions encode_hex and decode_hex transform strings and blobs to/from
their hexadecimal byte representation.
Port Contexts to TQL2
Section titled “Port Contexts to TQL2”The contexts feature is now available in TQL2. It has undergone significant
changes to make use of TQL2’s more powerful expressions. Contexts are shared
between TQL1 and TQL2 pipelines. All operators are grouped in the context
module, including the enrich and show contexts operators, which are now
called context::enrich and context::list, respectively. To create a new
context, use the context::create_lookup_table, context::create_bloom_filter,
or context::create_geoip operators.
Lookup table contexts now support separate create, write, and read timeouts via
the create_timeout, write_timeout, and read_timeout options, respectively.
The options are exclusive to contexts updated with TQL2’s context::update
operator.
By @dominiklohmann in #4753.
Improve to_splunk TLS functionality
Section titled “Improve to_splunk TLS functionality”The to_splunk operator now supports the cacert, certfile, and keyfile
options to provide certificates for the TLS connection.
Implement --limit flag for the chart operator
Section titled “Implement --limit flag for the chart operator”The --limit option for the TQL1 chart operator controls the previously
hardcoded upper limit on the number of events in a chart. The option defaults
to 10,000 events.
By @IyeOnline in #4757.
Add parse_time and format_time methods
Section titled “Add parse_time and format_time methods”The new parse_time and format_time functions transform strings into
timestamps and vice versa.
Implement x[y] record indexing
Section titled “Implement x[y] record indexing”Indexing records with string expressions is now supported.
Implement split, split_regex, and join
Section titled “Implement split, split_regex, and join”The split and split_regex functions split a string into a list of strings
based on a delimiter or a regular expression, respectively.
The join aggregation function concatenates a strings into a single string,
optionally separated by a delimiter.
By @dominiklohmann in #4799.
Fix crash in context::enrich for heterogeneous enrichments
Section titled “Fix crash in context::enrich for heterogeneous enrichments”The network function returns the network address of a CIDR subnet. For
example, 192.168.0.0/16.network() returns 192.168.0.0.
By @dominiklohmann in #4828.
Introduce a zip function for merging lists
Section titled “Introduce a zip function for merging lists”The zip function merges two lists into a single list of a record with two
fields left and right. For example, zip([1, 2], [3, 4]) returns [{left: 1, right: 3}, {left: 2, right: 4}].
By @dominiklohmann in #4803.
PRs 4716-4807
Section titled “PRs 4716-4807”The following operators are now available in TQL2 for loading and
saving: load_amqp, save_amqp, load_ftp, save_ftp, load_nic,
load_s3, save_s3, load_sqs, save_sqs, load_udp, save_udp,
load_zmq, save_zmq, save_tcp and save_email.
The following new operators are available in TQL2 to convert event
streams to byte streams in various formats: write_csv, write_feather,
write_json, write_lines, write_ndjson, write_parquet, write_pcap, write_ssv, write_tsv,
write_xsv, write_yaml, write_zeek_tsv.
Allow aggregation functions to be called on lists
Section titled “Allow aggregation functions to be called on lists”Aggregation functions now work on lists. For example, [1, 2, 3].sum() will
return 6, and ["foo", "bar", "baz"].map(x, x == "bar").any() will return
true.
By @dominiklohmann in #4821.
Port unordered, local, and remote to TQL2
Section titled “Port unordered, local, and remote to TQL2”The local and remote operators allow for overriding the location of a
pipeline. Local operators prefer running at a client tenzir process, and
remote operators prefer running at a remote tenzir-node process. These
operators are primarily intended for testing purposes.
The unordered operator throws away the order of events in a pipeline. This
causes some operators to run faster, e.g., read_ndjson is able to parse events
out of order through this. This operator is primarily intended for testing
purposes, as most of the time the ordering requirements are inferred from
subsequent operators in the pipeline.
By @dominiklohmann in #4835.
Changes
Section titled “Changes”Stop URL-encoding pub/sub topics
Section titled “Stop URL-encoding pub/sub topics”The topics provided to the publish and subscribe operators now exactly match
the topic field in the corresponding metrics.
Using publish and subscribe without an explicitly provided topic now uses
the topic main as opposed to an implementation-defined special name.
By @dominiklohmann in #4738.
Finish porting loaders, printers and savers
Section titled “Finish porting loaders, printers and savers”The option ndjson for write_json operator has been removed in favor of a new
operator write_ndjson.
PRs 4741-4746
Section titled “PRs 4741-4746”The functions ocsf_category_name, ocsf_category_uid, ocsf_class_name, and
ocsf_class_uid are now called ocsf::category_name, ocsf::category_uid,
ocsf::class_name, and ocsf::class_uid, respectively. Similarly, the
package_add, package_remove, packages, and show pipelines operators are
now called package::add, package::remove, package::list, and
pipeline::list, respectively.
Improve names for the cache operator’s timeout options
Section titled “Improve names for the cache operator’s timeout options”The cache operator’s ttl and max_ttl options are now called read_timeout
and write_timeout, respectively.
By @dominiklohmann in #4758.
Prepare small fixes for release
Section titled “Prepare small fixes for release”The new string function now replaces the str function. The older str
name will be available as an alias for some time for compatibility but will
be removed in a future release.
Align argument parser usage format with docs
Section titled “Align argument parser usage format with docs”The usage string that is reported when an operator or function is being used incorrectly now uses the same format as the documentation.
Improve to_splunk TLS functionality
Section titled “Improve to_splunk TLS functionality”The tls_no_verify option of the to_splunk operator is now called
skip_peer_verification.
Bug Fixes
Section titled “Bug Fixes”Add timeout to multiline syslog
Section titled “Add timeout to multiline syslog”We fixed an oversight in the syslog parsers, which caused it to not yield an event until the next line came in.
By @IyeOnline in #4829.
Port Loaders, Printers, Savers
Section titled “Port Loaders, Printers, Savers”The docs for the sqs connector now correctly reflect the default of 3s for
the --poll-time option.
Ignore whole line when NDJSON parser fails
Section titled “Ignore whole line when NDJSON parser fails”The read_ndjson operator no longer uses an error-prone mechanism to continue
parsing an NDJSON line that contains an error. Instead, the entire line is
skipped.
Introduce {package,pipeline}::list
Section titled “Introduce {package,pipeline}::list”context inspect crashed when used to inspect a context that was previously
updated with context update with an input containing a field of type enum.
This no longer happens.
By @dominiklohmann in #4746.
Don’t allow manual erasing of contexts from packages
Section titled “Don’t allow manual erasing of contexts from packages”It is no longer possible to manually remove contexts that are installed as part of a package.
Fix ODR violation of tenzir::socket type
Section titled “Fix ODR violation of tenzir::socket type”The TQL1 and TQL2 sockets operators no longer crash on specific builds.
Improve to_splunk TLS functionality
Section titled “Improve to_splunk TLS functionality”The max_content_length option for the to_splunk operator was named incorrectly in
an earlier version to send_timeout. This has now been fixed.
Prepend the field added by enumerate
Section titled “Prepend the field added by enumerate”The enumerate operator now correctly prepends the added index field instead of
appending it.
By @dominiklohmann in #4756.
Add missing co_yields in save_http
Section titled “Add missing co_yields in save_http”The TQL2 save_http operator had a bug causing it to fail to connect and
get stuck in an infinite loop. This is now fixed and works as expected.
Fix str function quotes
Section titled “Fix str function quotes”The str function no longer adds extra quotes when given a string. For example,
str("") == "\"\"" was changed to str("") == "".
Port Contexts to TQL2
Section titled “Port Contexts to TQL2”The last metric emitted for each run of the enrich operator was incorrectly
named tenzir.enrich.metrics instead of tenzir.metrics.enrich, causing it not
to be available via metrics enrich.
By @dominiklohmann in #4753.
Make to_hive a “local” operator
Section titled “Make to_hive a “local” operator”The to_hive operator now correctly writes files relative to the working
directory of a tenzir client process instead of relative to the node.
By @dominiklohmann in #4771.