Zeek
The Zeek network monitor translates raw packets into structured logs.
Tenzir supports various Zeek use cases, such as continuous ingestion, ad-hoc log file processing, and even generating Zeek logs.
We wrote several Zeek blog posts in the past that cover various aspects of the Zeek integration in much more detail.
Zeek logs come in three forms in practice, all of which Tenzir can parse natively:
zeek-tsv
: Tab-Separated Values (TSV) with a custom header.zeek-json
: One NDJSON file for all log types (aka. JSON Streaming) including an extra_path
and_write_ts
field.json
: One NDJSON file per log type.
Ingest logs into a node
To ingest Zeek logs into a Tenzir node, you have multiple options.
Easy-button import with the official Zeek package
Our official Zeek package makes it easy to ship your Zeek logs to a Tenzir node. Install the package first:
Then add this to your $PREFIX/share/zeek/site/local.zeek
to send all logs that
Zeek produces to a Tenzir node:
For ad-hoc command line processing you can also pass tenzir/import
to a Zeek
invocation:
For further details on how to use our Zeek package read our blog post Native Zeek Log Rotation & Shipping.
Run an import pipeline when rotating logs
If you cannot use our Zeek package, it is still possible to let Zeek trigger an import pipeline upon rotation. Zeek's logging framework can execute a shell script whenever it rotates a log file.
This requires setting Log::default_rotation_interval
to a non-zero value. The
default of 0 secs
means that log rotation is disabled. Add this to
$PREFIX/share/zeek/site/local.zeek
, which is the place for local configuration
changes:
redef Log::default_rotation_interval = 1 day;
Then redefine
Log::default_rotation_postprocessor_cmd
to point to your shell script, e.g., /usr/local/bin/ingest
:
redef Log::default_rotation_postprocessor_cmd=/usr/local/bin/ingest;
Here is an example ingest
script that imports all files into a Tenzir node:
Our blog post Native Zeek Log Rotation & Shipping provides further details on this method.
Run Zeek on a packet pipeline
You can run Zeek on a pipeline of PCAP packets and continue processing the logs
in the same pipeline. A stock Tenzir installation comes with a
user-defined zeek
operator that looks
as follows:
This allows you run Zeek on a packet trace as follows:
You can also perform more elaborate packet filtering by going through the
pcap
parser:
Read the in-depth blog
post for more
details about the inner workings of the shell
operator.
Process logs with a pipeline on the command line
Zeek ships with a helper utility zeek-cut
that operators on Zeek's
tab-separated logs. For example, to extract the host pairs from a conn log:
The list of arguments to zeek-cut
are the column names of the log. The
select
operator performs the equivalent in Tenzir
after we parse the logs as zeek-tsv
:
Since pipelines are multi-schema and the Zeek TSV parser is aware of log boundaries, you can also concatenate logs of various types:
Generate Zeek TSV from arbitrary data
The zeek-tsv
is not only a parser, but also a
printer. This means you can render any data as Zeek TSV log.
For example, print the Tenzir version as Zeek TSV log:
This yields the following output:
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path tenzir.version
#open 2023-12-16-08-47-12.372511
#fields version major minor patch tweak
#types string count count count count
v4.6.4-155-g0b75e93026 4 6 4 155
#close 2023-12-16-08-47-12.372736