Zeek

Zeek Logo

The Zeek network security monitor is the swiss-army knife for network protocol analysis. It generates detailed logs that provide an in-depth view over protocol activity and security-related events. VAST has first-class support for Zeek, both on the import and export side.

A brief historical digression: VAST’s data-model closely resembles Zeek’s type system because the first prototype of VAST was purpose-built to process only Zeek data.

Import

Zeek’s logging framework supports multiple output formats, with the most common ones being TSV and JSON. In the following, we describe how to import these formats.

The documentation for the zeek import format describes the command usage in more technical detail.

Tab-Separated Values (TSV)

The TSV format is the default output format of Zeek. For each log type, Zeek creates one file (e.g., conn.log, http.log, notice.log). A TSV file includes a header that describes the field names and types, followed by bulk of the data where each line represents one log entry in tab-separated form.

VAST’s Zeek parser infers the schema from the header, which makes it easy to process TSV data:

vast import zeek < conn.log

It is also possible to concatenate multiple log types into a single stream, because VAST recreates its internal parser whenever it encounters anew header in the input stream. This facilitates bulk import of logs:

zcat *.log.gz | vast import zeek

JSON

All JSON output from Zeek is line-delimited JSON. There exist two output modes: one log type per file and a single stream with all log types.

Native

Zeek’s default JSON output mode generates one file per log type. (This is the same as TSV) . Enabling this mode requires loading the script tuning/json-logs, by either passing it the command line or adding it to PREFIX/site/local.zeek.

Unlike the TSV, the JSON format does not contain type information. This loss of typing reduces the expressiveness of the data, e.g., IP addresses and plain strings are now indistinguishable. VAST reconstructs the lost type information by mapping field names to types in a schema, which re-enables type-specific query operations, such as performing top-k prefixes searches on IP addresses.

Import a JSON conn.log requires passing the type explicitly via -t zeek.conn:

vast import json -t zeek.conn < conn.log

Corelight

Corelight offers a Zeek package that switches from one-file-per-type logging to a single stream. In this configuration, Zeek writes out log events as they occur. Because log events of different types can now show up interleaved in the output, a new field _path contains the type of the log, such as conn or http.

VAST has native support for processing streaming JSON:

vast import corelight-json < zeek.log

Export

Because the data model of Zeek and VAST are very similar, VAST can render results in TSV format, for example:

vast export zeek '#type == "zeek.conn" && (6.6.6.6 || orig_bytes > 10kB)'

The documentation for the zeek export format describes the command usage in more technical detail.