File

Tenzir supports reading from and writing to files, including non-regular files, such as Unix domain sockets, standard input, standard output, and standard error.

When ~ is the first character in the file path, the operator substitutes it with the $HOME environment variable.

URL Support

The URL scheme file:// dispatches to load_file and save_file for seamless URL-style use via from and to.

Examples

Read a file

Read from a file and parse it in the format applied by the file extension:

from "/tmp/file.json"

The from operator automatically decompresses the file, if the suffix list contains a supported compression algorithm:

from "/tmp/file.json.gz"

Some operators perform better when the entire file arrives as a single block of bytes, such as the yara operator. In this case, passing mmap=true runs more efficiently:

from "/sandbox/malware.gz", mmap=true {
  decompress "gzip"
  yara "rule.yaml"
}

Follow a file

A pipeline typically completes once it reads the end of a file. Pass follow=true to disable this behavior and instead wait for new data written to it. This is similar to running tail -f on a file.

from "/tmp/never-ending-stream.ndjson", follow=true

Write a file

Write to a file in the format implied by the file extension:

version
to "/tmp/tenzir-version.json"

The to operator automatically compresses the file, if the suffix list contains a supported compression algorithm:

version
to "/tmp/tenzir-version.json.bz2"

Append to a file

In case the file exists and you do not want to overwrite it, pass append=true as option:

from {x: 42}
to "/tmp/event.csv", append=true

Read/write a Unix domain socket

Pass uds=true to signal that the file is a Unix domain socket:

to "/tmp/socket", uds=true {
  write_ndjson
}

When reading from a Unix domain socket, Tenzir automatically figures out whether the file is regular or a socket:

from "/tmp/socket" {
  read_ndjson
}