Skip to main content
Version: Next

read_csv

Read CSV (Comma-Separated Values) from a byte stream.

read_csv [list_sep=str, null_value=str, comments=bool, header=str, auto_expand=bool,
          schema=str, selector=str, schema_only=bool, raw=bool, unflatten=str]

Description​

The read_csv operator transforms a byte stream into a event stream by parsing the bytes as CSV.

auto_expand = bool (optional)​

Automatically add fields to the schema when encountering events with too many values instead of dropping the excess values.

comments = bool (optional)​

Treat lines beginning with "#" as comments.

header = str (optional)​

The string to be used as a header for the parsed values. If unspecified, the first line of the input is used as the header.

list_sep = str (optional)​

The string separating the elements inside a list.

Defaults to ;.

null_value = str (optional)​

The string denoting an absent value.

Defaults to empty string ("").

raw = bool (optional)​

Use only the raw types that are native to the parsed format. Fields that have a type specified in the chosen schema will still be parsed according to the schema.

In the case of CSV, this means that no parsing of data takes place at all and every value remains a string, unless the field is in the schema.

schema = str (optional)​

Provide the name of a schema to be used by the parser.

If a schema with a matching name is installed, the result will always have all fields from that schema.

  • Fields that are specified in the schema, but did not appear in the input will be null.
  • Fields that appear in the input, but not in the schema will also be kept. schema_only=true can be used to reject fields that are not in the schema.

If the given schema does not exist, this option instead assigns the output schema name only.

The schema option is incompatible with the selector option.

selector = str (optional)​

Designates a field value as schema name with an optional dot-separated prefix.

The string is parsed as <filename>[:<prefix>]. The prefix is optional and will be prepended to the field value to generate the schema name.

For example, the Suricata EVE JSON format includes a field event_type that contains the event type. Setting the selector to event_type:suricata causes an event with the value flow for the field event_type to map onto the schema suricata.flow.

The selector option is incompatible with the schema option.

schema_only = bool (optional)​

When working with an existing schema, this option will ensure that the output schema has only the fields from that schema. If the schema name is obtained via a selector and it does not exist, this has no effect.

This option requires either schema or selector to be set.

unflatten = str (optional)​

A delimiter that, if present in keys, causes values to be treated as values of nested records.

A popular example of this is the Zeek JSON format. It includes the fields id.orig_h, id.orig_p, id.resp_h, and id.resp_p at the top-level. The data is best modeled as an id record with four nested fields orig_h, orig_p, resp_h, and resp_p.

Without an unflatten separator, the data looks like this:

Without unflattening
{
  "id.orig_h": "1.1.1.1",
  "id.orig_p": 10,
  "id.resp_h": "1.1.1.2",
  "id.resp_p": 5
}

With the unflatten separator set to ., Tenzir reads the events like this:

With 'unflatten'
{
  "id": {
    "orig_h": "1.1.1.1",
    "orig_p": 10,
    "resp_h": "1.1.1.2",
    "resp_p": 5
  }
}

Examples​

Header in input​

input.csv
message,count,ip
some text,42,"1.1.1.1"
more text,100,"1.1.1.2"
load "input.csv"
read_csv
―――――――――――――――――――――――――――――――――――――――――――――――――
{ message: "some text", count: 42, ip: 1.1.1.1 }
{ message: "more text", count: 100, ip: 1.1.1.2 }

Manually specified header​

input_no_header.csv
some text,42,"1.1.1.1"
more text,100,"1.1.1.2"
load "input_no_header.csv"
read_csv header="message,count,ip"
―――――――――――――――――――――――――――――――――――――――――――――――――
{ message: "some text", count: 42, ip: 1.1.1.1 }
{ message: "more text", count: 100, ip: 1.1.1.2 }