The import command ingests data. An optional filter expression allows for
restricing the input to matching events. The format of the imported data must
be explicitly specified:
vast import [options] <format> [options] [expr]
The import command is the dual to the export command.
The --type / -t option filters known event types based on a prefix. E.g.,
vast import json -t zeek matches all event types that begin with zeek, and
restricts the event types known to the import command accordingly.
VAST permanently tracks imported event types. They do not need to be specified
again for consecutive imports.
The netflow import listens for Flexible NetFlow messages (i.e., NetFlow v5,
NetFlow v9, or IPFIX).
Please note that the NetFlow formats are only supported directly, not the thin
nffile wrapper from the nfdump-family of tools. For compatibility with
nffile, the nfreplay tool can be used:
# listen on :9995/udp
vast import netflow -l :9995/udp
# replay NetFlow v5 records to :9995/udp
nfreplay -v 5 < path/to/sample.nfcapd
VAST automatically calculates the Community
ID for every NetFlow record for
better pivoting support. The extra computation induces an overhead of
approximately 10% of the ingestion rate. The option --disable-community-id can
be used to disable the computation completely.
For example, to filter records by community ID, you can use the query
expression:
The community_id field exists in all NetFlow record types and facilitates
pivoting to PCAP, Zeek, Suricata, and other data formats that come with
community ID support.
The PCAP import format uses libpcap to read network
packets from a trace or an interface.
VAST automatically calculates the Community
ID for PCAPs for better
pivoting support. The extra computation induces an overhead of approximately 15%
of the ingestion rate. The option --disable-community-id can be used to
disable the computation completely.
The suricata import format consumes
EVE
JSON logs from Suricata. EVE is output is
Suricata's unified format to log all types of activity as single stream of
line-delimited JSON.
For each log entry, VAST parses the field event_type to determine the
specific record type and then parses the data according to the known schema.
The json import format consumes line-delimited
JSON objects
according to a specified schema. That is, one line corresponds to one event.
The object field names correspond to record field names.
JSON's can express only a subset VAST's data model. For example, VAST has
first-class support IP addresses but JSON can only represent them as strings.
To get the most out of your data, it is therefore important to define a schema
to get a differentiated view of the data.
The infer command also supports schema inference for JSON data. For example,
head data.json | vast infer will print a raw schema that can be supplied to
--schema-file / -s as file or to --schema / -S as string. However, after
infer dumps the schema, the generic type name should still be adjusted and
this would be the time to annotate fields with additional attributes, such as
#timestamp or #skip.
If no type prefix is specified with --type / -t, or multiple types match
based on the prefix, VAST uses an exact match based on the field names to
automatically deduce the event type for every line in the input.
The CSV import format
consumes comma-separated values in tabular form. The first line in a CSV file
must contain a header that describes the field names. The remaining lines
contain concrete values. Except for the header, one line corresponds to one
event.
Because CSV has no notion of typing, it is necessary to select a layout via
--type/-t whose field names correspond to the CSV header field names.
1258532143.457078 xvWLhxgUmj5 192.168.1.102 1170 192.168.1.1 53 udp dns 0.068511 36 215 SF - 0 Dd 1 64 1 243 (empty)
1258532203.657268 feNcvrZfDbf 192.168.1.104 1174 192.168.1.1 53 udp dns 0.170962 36 215 SF - 0 Dd 1 64 1 243 (empty)
1258532331.365294 aLsTcZJHAwa 192.168.1.1 5353 224.0.0.251 5353 udp dns 0.100381 273 0 S0 - 0 D 2 329 0 0 (empty)
When Zeek rotates
logs, it
produces compressed batches of *.tar.gz regularly. Ingesting a compressed
batch involves unpacking and concatenating the input before sending it to VAST: