Did you know that Zeek supports log rotation triggers, so that you can do anything you want with a newly rotated batch of logs?
This blog post shows you how to use Zeek's native log rotation feature to conveniently invoke any post-processor, such as a log shipper. In our examples we show how to to ingest data into Tenzir, but you can plug in any downstream tooling.
External Log Shipping (pull)
In case you're not using Zeek's native log rotation trigger, you may observe a directory to which Zeek periodically writes files. For example, the utility zeek-archiver does that.
Generic log shippers can take care of that as well. Your mileage may vary. For example, Filebeat works for stock Zeek only. The parsing logic is hard-coded for every log type. If you have custom scripts or extend some logs, you're left alone. Filebeat also uses the stock Zeek JSON output, which has no type information. Filebeat then brings the typing back manually later as it converts the logs to the Elastic Common Schema (ECS).
Native Log Shipping (push)
There's also a lesser known, push-based option using Zeek's logging framework. You can provide a shell script that Zeek invokes whenever it rotates a file. The shell script receives the filename of the rotated file plus some additional metadata as arguments.
First, to activate log rotation, you need to set
Log::default_rotation_interval to a non-zero value. The default of
means that log rotation is disabled.
Second, to customize what's happening on rotation you can redefine
to point to a shell script.
For example, to rotate all log files every 10 minutes with a custom
script, you can invoke Zeek as follows:
zeek -r trace.pcap \
Let's take a look at this
ingest shell script in more detail. Zeek always
passes 6 arguments to the post-processing script:
- The filename of the log, e.g.,
- The type of the log (aka.
path), such as
- Timestamp when Zeek opened the log file
- Timestamp when Zeek closed (= rotated) the log file
- A flag that is true when rotation occurred due to Zeek terminating
- The format of the log, which is either
Here's a complete example that uses (1), (2), and (6):
if [ "$writer" = "ascii" ]; then
elif [ "$writer" = "json" ]; then
format="json --schema zeek.$base_name"
echo "unsupported Zeek writer: $writer"
pipeline="from file $file_name read $format | import"
Post-processing with Tenzir pipelines
When you run Zeek as above, the
ingest script dynamically constructs an
ingestion pipeline based on the type of the Zeek log at hand. Given your logging
format (TSV or JSON), the pipelines for a rotated
conn.log file may look like
from file /path/to/conn.log read zeek-tsv | import
from file /path/to/conn.log read json --schema zeek.conn | import
pipeline="from file $file_name read $format \
| extend filename=$file_name \
If you want post-processing with Tenzir pipelines out of the box, use our official Zeek package:
zkg install zeek-tenzir
After installing the package, you have two options to run pipelines on rotated Zeek logs:
tenzir-importZeek script to ship logs to a local Tenzir node
# Start a node.
# Ship logs to it and delete the original files.
zeek -r trace.pcap tenzir/import
zeekto keep the original logs.
Write Zeek scripts to register pipelines manually:
# Activate log rotation by setting a non-zero value.
redef Log::default_rotation_interval = 10 mins;
Tenzir::postprocess("to directory /tmp/logs write parquet");
The above Zeek script hooks up two pipelines via the function
Tenzir::postprocess. Each pipeline executes upon log rotation and receives the Zeek log file as input. The first imports all data via
importand the second writes the logs as
Zeek implements the log rotation logic by spawning a separate child process. When the (parent) Zeek process dies, the children become orphaned and keep running until completion.
The implication is that Zeek cannot re-trigger a failed post-processing command. So you have exactly one shot. This may not be a problem for trace file analysis, but live deployments may require higher reliability guarantees. For such scenarios, we recommend to use the post-processing script as a notifier, e.g., to signal another tool that it can now process a file.
For ultimate control over logging, you can always develop your own writer plugin that immediately ship logs instead of going through the file system.
This blog post shows how you can use Zeek's native log rotation feature to invoke an arbitrary command as soon as a log file gets rotated. This approach provides an attractive alternative that turns pull-based file monitoring into more flexible push-based delivery.
If you are looking for an efficient way to get your Zeek logs flowing, give Tenzir a try. Our Zeek package makes it easy to launch post-processing pipelines natively from Zeek. And don't forget to check out our other Zeek blogs.