🚀 Features
Section titled “🚀 Features”Implement a RabbitMQ-based AMQP connector
Section titled “Implement a RabbitMQ-based AMQP connector”The new amqp connector enables interaction with an AMQP 0-9-1 exchange,
supporting working with messages as producer (saver) and consumer (loader).
Add a —live option to the export operator
Section titled “Add a —live option to the export operator”The export operator now has a --live option to continuously emit events as
they are imported instead of those that already reside in the database.
Implement extended deployment options
Section titled “Implement extended deployment options”Nov 6, 2023 · @Dakostu · #3585
The new ttl_expires_in_ns shows the remaining time to live for a pipeline in
the pipeline manager.
Add a yara operator
Section titled “Add a yara operator”The new yara operator matches Yara rules on byte streams, producing structured
events when rules match.
Add show serves for debugging the serve operator
Section titled “Add show serves for debugging the serve operator”Oct 24, 2023 · @dominiklohmann · #3596
show serves displays all currently active serve IDs in the /serve API
endpoint, showing an overview of active pipelines with an on-demand API.
Add blob type for arbitrary binary data
Section titled “Add blob type for arbitrary binary data”Oct 23, 2023 · @jachris · #3581
The new blob type can be used to represent arbitrary binary data.
Add events field to output of show partitions
Section titled “Add events field to output of show partitions”Oct 18, 2023 · @dominiklohmann · #3580
The output of show partitions includes a new events field that shows the
number of events kept in that partition. E.g., the pipeline show partitions | summarize events=sum(events) by schema shows the number of events per schema
stored at the node.
Update the plugins submodule pointer to include extended & serialized pipeline states
Section titled “Update the plugins submodule pointer to include extended & serialized pipeline states”Oct 16, 2023 · @Dakostu · #3554
The new completed pipeline state in the pipeline manager shows when a
pipeline has finished execution.
If the node with running pipelines crashes, they will be marked as failed
upon restarting.
Add a velociraptor operator
Section titled “Add a velociraptor operator”The new velociraptor source supports submitting VQL queries to a Velociraptor
server. The operator communicates with the server via gRPC using a mutually
authenticated and encrypted connection with client certificates. For example,
velociraptor -q "select * from pslist()" lists processes and their running
binaries.
🔧 Changes
Section titled “🔧 Changes”Implement extended deployment options
Section titled “Implement extended deployment options”Nov 6, 2023 · @Dakostu · #3585
The new autostart and autodelete parameters for the pipeline manager
supersede the start_when_created and restart_with_node parameters and
extend restarting and deletion possibilities for pipelines.
Add blob type for arbitrary binary data
Section titled “Add blob type for arbitrary binary data”Oct 23, 2023 · @jachris · #3581
The string type is now restricted to valid UTF-8 strings. Use blob for
arbitrary binary data.
🐞 Bug Fixes
Section titled “🐞 Bug Fixes”Fix parsing of subnet columns in zeek-tsv
Section titled “Fix parsing of subnet columns in zeek-tsv”The zeek-tsv parser is now able to handle fields of type subnet correctly.
Fix predicate pushdown in export and other small fixes
Section titled “Fix predicate pushdown in export and other small fixes”Oct 26, 2023 · @dominiklohmann · #3599
A regression in Tenzir v4.3 caused exports to often consider all partitions as
candidates. Pipelines of the form export | where <expr> now work as expected
again and only load relevant partitions from disk.
The long option --skip-empty for read lines now works as documented.
Fix serve exiting prematurely
Section titled “Fix serve exiting prematurely”Oct 19, 2023 · @dominiklohmann · #3562
Pipelines ending with the serve operator no longer incorrectly exit 60 seconds
after transferring all events to the /serve endpoint, but rather wait until
all events were fetched from the endpoint.
Shutting down a node immediately after starting it now no longer waits for all partitions to be loaded.
Upgrade remaining usages of the adaptive table slice builder
Section titled “Upgrade remaining usages of the adaptive table slice builder”Oct 18, 2023 · @jachris · #3582
The csv parsed (or more generally, the xsv parser) now attempts to parse
fields in order to infer their types.
Check for duplicate field names in zeek_tsv_parser
Section titled “Check for duplicate field names in zeek_tsv_parser”Oct 18, 2023 · @eliaskosunen · #3578
Having duplicate field names in zeek-tsv data no longer causes a crash,
but rather errors out gracefully.
Detect and report incomplete object in JSON parser
Section titled “Detect and report incomplete object in JSON parser”Oct 16, 2023 · @jachris · #3570
When using read json, incomplete objects (e.g., due to truncated files) are
now reported as an error instead of silently discarded.