The Zeek network security monitor is the swiss-army knife for network protocol analysis. It generates detailed logs that provide an in-depth view over protocol activity and security-related events. VAST has first-class support for Zeek, both on the import and export side.
A brief historical digression: VAST’s data-model closely resembles Zeek’s type system because the first prototype of VAST was purpose-built to process only Zeek data.
Zeek’s logging framework supports multiple output formats, with the most common ones being TSV and JSON. In the following, we describe how to import these formats.
The documentation for the zeek import format describes the command usage in more technical detail.
The TSV format is the default output format of Zeek. For each log type, Zeek
creates one file (e.g.,
notice.log). A TSV file
includes a header that describes the field names and types, followed by bulk of
the data where each line represents one log entry in tab-separated form.
VAST’s Zeek parser infers the schema from the header, which makes it easy to process TSV data:
vast import zeek < conn.log
It is also possible to concatenate multiple log types into a single stream, because VAST recreates its internal parser whenever it encounters anew header in the input stream. This facilitates bulk import of logs:
zcat *.log.gz | vast import zeek
Zeek’s default JSON output mode generates one file per log type. (This is the
same as TSV) . Enabling this mode requires loading the script
tuning/json-logs, by either passing it the command line or adding it to
Unlike the TSV, the JSON format does not contain type information. This loss of typing reduces the expressiveness of the data, e.g., IP addresses and plain strings are now indistinguishable. VAST reconstructs the lost type information by mapping field names to types in a schema, which re-enables type-specific query operations, such as performing top-k prefixes searches on IP addresses.
Import a JSON
conn.log requires passing the type explicitly via
vast import json -t zeek.conn < conn.log
Corelight offers a Zeek
package that switches from
one-file-per-type logging to a single stream. In this configuration, Zeek
writes out log events as they occur. Because log events of different types can
now show up interleaved in the output, a new field
_path contains the type of
the log, such as
VAST has native support for processing streaming JSON:
vast import corelight-json < zeek.log
Because the data model of Zeek and VAST are very similar, VAST can render results in TSV format, for example:
vast export zeek '#type == "zeek.conn" && (188.8.131.52 || orig_bytes > 10kB)'
The documentation for the zeek export format describes the command usage in more technical detail.