🚀 Features
Section titled “🚀 Features”Introduce a zip function for merging lists
Section titled “Introduce a zip function for merging lists”Dec 3, 2024 · @dominiklohmann · #4803
The zip function merges two lists into a single list of a record with two
fields left and right. For example, zip([1, 2], [3, 4]) returns [{left: 1, right: 3}, {left: 2, right: 4}].
Port unordered, local, and remote to TQL2
Section titled “Port unordered, local, and remote to TQL2”Dec 3, 2024 · @dominiklohmann · #4835
The local and remote operators allow for overriding the location of a
pipeline. Local operators prefer running at a client tenzir process, and
remote operators prefer running at a remote tenzir-node process. These
operators are primarily intended for testing purposes.
The unordered operator throws away the order of events in a pipeline. This
causes some operators to run faster, e.g., read_ndjson is able to parse events
out of order through this. This operator is primarily intended for testing
purposes, as most of the time the ordering requirements are inferred from
subsequent operators in the pipeline.
Fix crash in context::enrich for heterogeneous enrichments
Section titled “Fix crash in context::enrich for heterogeneous enrichments”Dec 2, 2024 · @dominiklohmann · #4828
The network function returns the network address of a CIDR subnet. For
example, 192.168.0.0/16.network() returns 192.168.0.0.
Improve to_splunk TLS functionality
Section titled “Improve to_splunk TLS functionality”The to_splunk operator now supports the cacert, certfile, and keyfile
options to provide certificates for the TLS connection.
Implement encode_hex() and decode_hex()
Section titled “Implement encode_hex() and decode_hex()”The functions encode_hex and decode_hex transform strings and blobs to/from
their hexadecimal byte representation.
Implement --limit flag for the chart operator
Section titled “Implement --limit flag for the chart operator”Nov 29, 2024 · @IyeOnline · #4757
The --limit option for the TQL1 chart operator controls the previously
hardcoded upper limit on the number of events in a chart. The option defaults
to 10,000 events.
Allow aggregation functions to be called on lists
Section titled “Allow aggregation functions to be called on lists”Nov 28, 2024 · @dominiklohmann · #4821
Aggregation functions now work on lists. For example, [1, 2, 3].sum() will
return 6, and ["foo", "bar", "baz"].map(x, x == "bar").any() will return
true.
Implement encode_base64 and decode_base64
Section titled “Implement encode_base64 and decode_base64”Nov 26, 2024 · @raxyte · #4806
The new functions encode_base64 and decode_base64 encode and
decode blobs and strings as Base64.
Implement otherwise(<expr>, <expr>)
Section titled “Implement otherwise(<expr>, <expr>)”Nov 25, 2024 · @raxyte · #4794
The function otherwise(primary:any, fallback:any) provides a simple way
to specify a fallback expression when the primary expression
evaluates to null.
Add parse_time and format_time methods
Section titled “Add parse_time and format_time methods”The new parse_time and format_time functions transform strings into
timestamps and vice versa.
Implement split, split_regex, and join
Section titled “Implement split, split_regex, and join”Nov 21, 2024 · @dominiklohmann · #4799
The split and split_regex functions split a string into a list of strings
based on a delimiter or a regular expression, respectively.
The join aggregation function concatenates a strings into a single string,
optionally separated by a delimiter.
Implement append, prepend, and concatenate
Section titled “Implement append, prepend, and concatenate”Nov 21, 2024 · @dominiklohmann · #4792
The new append, prepend, and concatenate functions add an element to the
end of a list, to the front of a list, and merge two lists, respectively.
xs.append(y) is equivalent to [...xs, y], xs.prepend(y) is equivalent to
[y, ...xs], and concatenate(xs, ys) is equivalent to [...xs, ..ys].
Implement x[y] record indexing
Section titled “Implement x[y] record indexing”Nov 20, 2024 · @raxyte · #4795
Indexing records with string expressions is now supported.
Implement where and map on lists
Section titled “Implement where and map on lists”Nov 20, 2024 · @dominiklohmann · #4788
The <list>.map(<capture>, <expression>) function replaces each value from
<list> with the value from <expression>. Within <expression>, the elements
are available as <capture>. For example, to add 5 to all elements in the list
xs, use xs = xs.map(x, x + 5).
The <list>.where(<capture>, <predicate>) removes all elements from <list>
for which the <predicate> evaluates to false. Within <predicate>, the
elements are available as <capture>. For example, to remove all elements
smaller than 3 from the list xs, use xs = xs.where(x, x >= 3).
Port Contexts to TQL2
Section titled “Port Contexts to TQL2”Nov 14, 2024 · @dominiklohmann · #4753
The contexts feature is now available in TQL2. It has undergone significant
changes to make use of TQL2’s more powerful expressions. Contexts are shared
between TQL1 and TQL2 pipelines. All operators are grouped in the context
module, including the enrich and show contexts operators, which are now
called context::enrich and context::list, respectively. To create a new
context, use the context::create_lookup_table, context::create_bloom_filter,
or context::create_geoip operators.
Lookup table contexts now support separate create, write, and read timeouts via
the create_timeout, write_timeout, and read_timeout options, respectively.
The options are exclusive to contexts updated with TQL2’s context::update
operator.
PRs 4716-4807
Section titled “PRs 4716-4807”Nov 11, 2024 · @raxyte · #4716
The following operators are now available in TQL2 for loading and
saving: load_amqp, save_amqp, load_ftp, save_ftp, load_nic,
load_s3, save_s3, load_sqs, save_sqs, load_udp, save_udp,
load_zmq, save_zmq, save_tcp and save_email.
The following new operators are available in TQL2 to convert event
streams to byte streams in various formats: write_csv, write_feather,
write_json, write_lines, write_ndjson, write_parquet, write_pcap, write_ssv, write_tsv,
write_xsv, write_yaml, write_zeek_tsv.
Support decapsulating SLL2 packets
Section titled “Support decapsulating SLL2 packets”The decapsulate function now handles SLL2 frames (Linux cooked capture
encapsulation).
Port unroll to TQL2
Section titled “Port unroll to TQL2”Nov 7, 2024 · @dominiklohmann · #4736
The unroll operator is now available in TQL2. It takes a field of type list,
and duplicates the surrounding event for every element of the list.
🔧 Changes
Section titled “🔧 Changes”Prepare small fixes for release
Section titled “Prepare small fixes for release”The new string function now replaces the str function. The older str
name will be available as an alias for some time for compatibility but will
be removed in a future release.
Improve to_splunk TLS functionality
Section titled “Improve to_splunk TLS functionality”The tls_no_verify option of the to_splunk operator is now called
skip_peer_verification.
Align argument parser usage format with docs
Section titled “Align argument parser usage format with docs”Nov 22, 2024 · @jachris · #4740
The usage string that is reported when an operator or function is being used incorrectly now uses the same format as the documentation.
Finish porting loaders, printers and savers
Section titled “Finish porting loaders, printers and savers”Nov 19, 2024 · @raxyte · #4762
The option ndjson for write_json operator has been removed in favor of a new
operator write_ndjson.
Improve names for the cache operator’s timeout options
Section titled “Improve names for the cache operator’s timeout options”Nov 13, 2024 · @dominiklohmann · #4758
The cache operator’s ttl and max_ttl options are now called read_timeout
and write_timeout, respectively.
PRs 4741-4746
Section titled “PRs 4741-4746”Nov 9, 2024 · @jachris · #4741
The functions ocsf_category_name, ocsf_category_uid, ocsf_class_name, and
ocsf_class_uid are now called ocsf::category_name, ocsf::category_uid,
ocsf::class_name, and ocsf::class_uid, respectively. Similarly, the
package_add, package_remove, packages, and show pipelines operators are
now called package::add, package::remove, package::list, and
pipeline::list, respectively.
Stop URL-encoding pub/sub topics
Section titled “Stop URL-encoding pub/sub topics”Nov 7, 2024 · @dominiklohmann · #4738
The topics provided to the publish and subscribe operators now exactly match
the topic field in the corresponding metrics.
Using publish and subscribe without an explicitly provided topic now uses
the topic main as opposed to an implementation-defined special name.
🐞 Bug Fixes
Section titled “🐞 Bug Fixes”Add timeout to multiline syslog
Section titled “Add timeout to multiline syslog”Dec 3, 2024 · @IyeOnline · #4829
We fixed an oversight in the syslog parsers, which caused it to not yield an event until the next line came in.
Add missing co_yields in save_http
Section titled “Add missing co_yields in save_http”The TQL2 save_http operator had a bug causing it to fail to connect and
get stuck in an infinite loop. This is now fixed and works as expected.
Improve to_splunk TLS functionality
Section titled “Improve to_splunk TLS functionality”The max_content_length option for the to_splunk operator was named incorrectly in
an earlier version to send_timeout. This has now been fixed.
Fix ODR violation of tenzir::socket type
Section titled “Fix ODR violation of tenzir::socket type”Nov 26, 2024 · @raxyte · #4816
The TQL1 and TQL2 sockets operators no longer crash on specific builds.
Fix str function quotes
Section titled “Fix str function quotes”Nov 22, 2024 · @jachris · #4809
The str function no longer adds extra quotes when given a string. For example,
str("") == "\"\"" was changed to str("") == "".
Ignore whole line when NDJSON parser fails
Section titled “Ignore whole line when NDJSON parser fails”Nov 21, 2024 · @jachris · #4801
The read_ndjson operator no longer uses an error-prone mechanism to continue
parsing an NDJSON line that contains an error. Instead, the entire line is
skipped.
Don’t allow manual erasing of contexts from packages
Section titled “Don’t allow manual erasing of contexts from packages”It is no longer possible to manually remove contexts that are installed as part of a package.
Make to_hive a “local” operator
Section titled “Make to_hive a “local” operator”Nov 14, 2024 · @dominiklohmann · #4771
The to_hive operator now correctly writes files relative to the working
directory of a tenzir client process instead of relative to the node.
Port Contexts to TQL2
Section titled “Port Contexts to TQL2”Nov 14, 2024 · @dominiklohmann · #4753
The last metric emitted for each run of the enrich operator was incorrectly
named tenzir.enrich.metrics instead of tenzir.metrics.enrich, causing it not
to be available via metrics enrich.
Prepend the field added by enumerate
Section titled “Prepend the field added by enumerate”Nov 12, 2024 · @dominiklohmann · #4756
The enumerate operator now correctly prepends the added index field instead of
appending it.
Port Loaders, Printers, Savers
Section titled “Port Loaders, Printers, Savers”Nov 11, 2024 · @raxyte · #4716
The docs for the sqs connector now correctly reflect the default of 3s for
the --poll-time option.
Introduce {package,pipeline}::list
Section titled “Introduce {package,pipeline}::list”Nov 10, 2024 · @dominiklohmann · #4746
context inspect crashed when used to inspect a context that was previously
updated with context update with an input containing a field of type enum.
This no longer happens.