This release improves the stability of pipelines by applying backpressure more effectively, ensuring upstream components slow down before overwhelming subscribers. It also enhances daily operations by improving parquet reliability, adding Base58 support, and extending the built-in OCSF operators and functions to support the newly released OCSF 1.7.0.
Download the release on GitHub.
Features
Section titled “Features”Support for OCSF v1.7.0
Section titled “Support for OCSF v1.7.0”Tenzir now supports the newly released OCSF version 1.7.0.
Improved Encoding and Decoding Support
Section titled “Improved Encoding and Decoding Support”We have added two new functions: encode_base58 and decode_base58. As the names
imply, these Base58 encode/decode string or blob values, just like the already
existing functions for Base64, Hex or URL encoding.
With this change, we also enabled the usage of all encoding and decoding functions
on secret values.
By @IyeOnline in #5572.
Changes
Section titled “Changes”Parse time to UNIX epoch
Section titled “Parse time to UNIX epoch”The parse_time() function would earlier default to 1900 when the year was
unspecified. This has now been changed to 1970 to match the assumptions about
epoch in other parts of the language.
Easier multi-key deduplication
Section titled “Easier multi-key deduplication”We now support multiple keys for deduplication, e.g., deduplicate a, b.c, d
is now equivalent to the previously unwieldy expression deduplicate {x: a, y: b.c, z: d}.
Bug Fixes
Section titled “Bug Fixes”Backpressure in publish and subscribe
Section titled “Backpressure in publish and subscribe”Previously, the backpressure mechanism in publish and subscribe was not
working as intended. Thus, publishing pipelines continued processing data
even when downstream consumers were lagging far behind. This is now fixed. As a
result, memory consumption for pipelines connected by publish and subscribe
is reduced significantly in those cases.
Gracefully handle server errors in the web plugin
Section titled “Gracefully handle server errors in the web plugin”The node now prints a proper error message in case the builtin web server can’t start because the port is already in use.
Improved parquet support
Section titled “Improved parquet support”We fixed a crash when performing some operations on data read from external parquet files, such as Amazon Security Lake.
By @IyeOnline in #5579.
Fixed a race in concurrent function application
Section titled “Fixed a race in concurrent function application”We fixed a race condition in case multiple pipelines used the same transformation on the same data, such as string manipulations in where clauses to filter the input. The bug caused those pipelines to fail sporadically.