Skip to content

from_file

Under Active Development

This operator is still under active development.

Reads one or multiple files from a filesystem.

from_file url:string, [watch=bool, remove=bool, path_field=field] { … }

The from_file operator reads files from local filesystems or cloud storage, with support for glob patterns, automatic format detection, and file monitoring.

URL or local filesystem path where data should be read from.

The characters * and ** have a special meaning. * matches everything except /. ** matches everything including /. The sequence /**/ can also match nothing. For example, foo/**/bar matches foo/bar.

The URL can include additional options. For s3://, the options that can be included in the URI as query parameters are region, scheme, endpoint_override, allow_bucket_creation, and allow_bucket_deletion. For gs://, the supported parameters are scheme, endpoint_override, and retry_limit_seconds.

In addition to processing all existing files, this option keeps the operator running, watching for new files that also match the given URL. Currently, this scans the filesystem up to every 10s.

Defaults to false.

Deletes files after they have been read completely.

Defaults to false.

This makes the operator insert the path to the file where an event originated from before emitting it.

By default, paths will not be inserted into the outgoing events.

Pipeline to use for parsing the file. By default, this pipeline is derived from the path of the file, and will not only handle parsing but also decompression if applicable. This is using the same logic as from.

from_file "s3://my-bucket/**.csv"

Read every .json file in /data as Suricata EVE JSON

Section titled “Read every .json file in /data as Suricata EVE JSON”
from_file "/data/**.json" {
read_suricata
}

Read all files from S3 continuously and delete them afterwards

Section titled “Read all files from S3 continuously and delete them afterwards”
from_file "s3://my-bucket/**", watch=true, remove=true

from, load_file

Last updated: