Download the release on GitHub.
Features
Section titled “Features”Add a —null option to the lines parser
Section titled “Add a —null option to the lines parser”The lines parser can now handle null delimited “lines” with the --null flag.
Support bytes inputs in the buffer operator
Section titled “Support bytes inputs in the buffer operator”The buffer operator now works with bytes inputs in addition to the existing
support for events inputs.
By @dominiklohmann in #4594.
Precise Parsing
Section titled “Precise Parsing”The CEF, CSV, GELF, JSON, KV, LEEF, Suricata, Syslog, XSV, YAML and Zeek JSON parsers now properly adhere to the schema of the read data. Previously, parsers would merge heterogeneous input into a single, growing schema, inserting nulls for fields that did not exist in some events.
The fluent-bit source now properly adheres to the schema of the read data.
The CEF, CSV, GELF, JSON, KV, LEEF, Suricata, Syslog, XSV, YAML and Zeek JSON
parsers now all support the --schema, --selector flags to parse their data
according to some given schema, as well as various other flags to more
precisely control their output schema.
By @IyeOnline in #4527.
Metrics for TCP connections
Section titled “Metrics for TCP connections”metrics tcp shows metrics for TCP connections, emitted once every second per
connection. The metrics contains the reads and writes on the socket and the
number of bytes transmitted.
Dynamically grow simdjson buffer if necessary
Section titled “Dynamically grow simdjson buffer if necessary”The JSON parser is now able to also handle extremely large events when not using the NDJSON or GELF mode.
By @IyeOnline in #4590.
Make the kv-parser consider quotes when looking for separators
Section titled “Make the kv-parser consider quotes when looking for separators”The kv parser now allows for keys and values to be enclosed in double quotes:
Split matches within quotes will not be considered. Quotes will be trimmed of
keys and values. For example "key"="nested = value, fun" will now successfully
parse as { "key" : "nested = value, fun" }.
By @IyeOnline in #4591.
Implement the azure-blob-storage connector
Section titled “Implement the azure-blob-storage connector”The new azure-blob-storage connector allows reading from and writing
to Azure Blob Storage via an URI.
By @IyeOnline in #4617.
Changes
Section titled “Changes”Prefer recent partitions for retro lookups
Section titled “Prefer recent partitions for retro lookups”The lookup operator now prefers recent data in searches for lookups against
historical data instead of using the order in which context updates arrive.
By @dominiklohmann in #4636.
Switch the index to basic messaging
Section titled “Switch the index to basic messaging”We removed the unused --snapshot option from the lookup operator.
Precise Parsing
Section titled “Precise Parsing”The JSON parser’s --precise option is now deprecated, as the “precise” mode
is the new default. Use --merge to get the previous “imprecise” behavior.
The JSON parser’s --no-infer option has been renamed to --schema-only. The
old name is deprecated and will be removed in the future.
By @IyeOnline in #4527.
Stabilize the bitz format
Section titled “Stabilize the bitz format”Tenzir’s internal wire format bitz is now considered stable. Note that the
format underwent significant changes as part of its stabilization, and is
incompatible with bitz from Tenzir Node v4.20 and older.
By @dominiklohmann in #4633.
Bug Fixes
Section titled “Bug Fixes”Stabilize the bitz format
Section titled “Stabilize the bitz format”We fixed a very rare crash in the zero-copy parser implementation of read feather and read parquet that was caused by releasing shared memory too
early.
By @dominiklohmann in #4633.
Keep from tcp pipelines running on connection failures
Section titled “Keep from tcp pipelines running on connection failures”Pipelines starting with from tcp no longer enter the failed state when an
error occurrs in one of the connections.
Stop using connection timeout to get node components
Section titled “Stop using connection timeout to get node components”The import and partitions operators and the tenzir-ctl rebuild command no
longer occasionally fail with request timeouts when the node is under high load.
By @dominiklohmann in #4597.
Make read json --arrays-of-objects faster
Section titled “Make read json --arrays-of-objects faster”We fixed an accidentally quadratic scaling with the number of top-level array
elements in read json --arrays-of-objects. As a result, using this option will
now be much faster.
By @dominiklohmann in #4601.
Precise Parsing
Section titled “Precise Parsing”We fixed various edge cases in parsers where values would not be properly parsed as typed data and were stored as plain text instead. No input data was lost, but no valuable type information was gained either.
By @IyeOnline in #4527.