Tenzir comes with a wide range of built-in pipeline operators.
Analyze
Section titled “Analyze”Charts
Section titled “Charts”Connecting Pipelines
Section titled “Connecting Pipelines”Contexts
Section titled “Contexts”Creates a Bloom filter context.
context::create_bloom_filter "ctx", capacity=1Mi, fp_probability=0.01Creates a GeoIP context.
context::create_geoip "ctx", db_path="GeoLite2-City.mmdb"Detection
Section titled “Detection”Encode & Decode
Section titled “Encode & Decode”Escape Hatches
Section titled “Escape Hatches”Executes a system command and hooks its stdin and stdout into the pipeline.
shell "echo hello"Filter
Section titled “Filter”Drops events and emits a warning if the invariant is violated.
assert name.starts_with("John")Emits a warning if the pipeline does not have the expected throughput
assert_throughput 1000, within=1sKeeps a range of events within the interval
[begin, end) stepping by stride.slice begin=10, end=30Flow Control
Section titled “Flow Control”Runs a pipeline periodically according to a cron expression.
cron "* */10 * * * MON-FRI" { from_http "https://example.org" }Spawns a subpipeline for every incoming event, with the event bound to
$this.each { from $this }Routes events with the same key through the same subpipeline.
group tenant { summarize count() }Runs a subpipeline across multiple parallel workers.
parallel 4 { parsed = data.parse_json() }Host Inspection
Section titled “Host Inspection”Inputs
Section titled “Inputs”Accepts incoming Elasticsearch-compatible Bulk API requests and forwards them as events.
accept_elasticsearch "0.0.0.0:9200"Accepts incoming HTTP requests and forwards them as events.
accept_http "0.0.0.0:8080" { read_json }Accepts incoming OpenSearch-compatible Bulk API requests and forwards them as events.
accept_opensearch "0.0.0.0:9200"Accepts incoming TCP or TLS connections and yields events.
accept_tcp "0.0.0.0:8090" { read_json }Listens on a ZeroMQ endpoint and receives events.
accept_zmq "tcp://0.0.0.0:5555", prefix="alerts/" { read_json }Reads events from Amazon CloudWatch.
from_amazon_cloudwatch "/aws/lambda/api", mode="search"Receives messages from an AMQP queue.
from_amqp "amqp://admin:pass@0.0.0.1:5672/vhost", queue="events"Reads one or multiple files from Azure Blob Storage.
from_azure_blob_storage "abfs://container/data/**.json"Downloads bytes via FTP and parses them with a subpipeline.
from_ftp "ftp.example.org/events.ndjson" { read_ndjson }Subscribes to a Google Cloud Pub/Sub subscription and yields events.
from_google_cloud_pubsub project_id="my-project", subscription_id="my-sub"Reads one or multiple files from Google Cloud Storage.
from_google_cloud_storage "gs://my-bucket/data/**.json"Sends an HTTP/1.1 request and returns the response as events.
from_http "https://example.com/api/events.json"Reads events from a Microsoft Graph collection.
from_microsoft_graph "auditLogs/signIns", auth={…}Reads events from a MySQL database.
from_mysql table="users", host="db.example.com", database="mydb"Retrieves PowerQuery results from SentinelOne Singularity Data Lake.
from_sentinelone_data_lake "https://…", …Connects to a remote TCP or TLS endpoint and receives events.
from_tcp "example.org:4000" { read_json}Submits VQL to a Velociraptor server and returns the response as events.
from_velociraptor subscribe="Windows"Connects to a remote ZeroMQ publisher and receives events.
from_zmq "tcp://collector.example.com:5555", prefix="alerts/" { read_json }Internals
Section titled “Internals”An in-memory buffer to improve handling of data spikes in upstream operators.
buffer 10M, policy="drop"Modify
Section titled “Modify”Performs DNS lookups to resolve IP addresses to hostnames or hostnames to IP addresses.
dns_lookup ip_address, result=dns_infoRemoves fields containing null values from the event.
drop_null_fields name, metadata.idMoves values from one field to another, removing the original field.
move id=parsed_id, ctx.message=incoming.statusAdjusts timestamps relative to a given start time, with an optional speedup.
timeshift ts, start=2020-01-01Returns a new event for each member of a list or a record in an event, duplicating the surrounding event.
unroll namesAutomatically assigns enum strings from their integer counterparts and vice versa.
ocsf::deriveOutputs
Section titled “Outputs”Starts an HTTP server and streams bytes produced by a nested pipeline to connected clients.
serve_http "0.0.0.0:8080" { write_ndjson }Listens for incoming TCP connections and sends events to all connected clients.
serve_tcp "0.0.0.0:8090" { write_json }Listens on a ZeroMQ endpoint and sends events.
serve_zmq "tcp://0.0.0.0:5555", encoding="json", prefix=f"{kind}/"Sends events to Amazon CloudWatch.
to_amazon_cloudwatch "/tenzir/events", stream="default"Writes events to one or multiple blobs in Azure Blob Storage.
to_azure_blob_storage "abfs://container/data/{uuid}.json" { write_ndjson }Sends events to the Microsoft Azure Logs Ingestion API.
to_azure_log_analytics tenant_id="...", workspace_id="..."Sends events to an Elasticsearch-compatible Bulk API.
to_elasticsearch "localhost:9200", …Writes events to one or multiple files on a filesystem.
to_file "/tmp/out.json" { write_ndjson }Prints events to bytes and uploads them via FTP.
to_ftp "ftp.example.org/events.ndjson" { write_ndjson }Publishes events to a Google Cloud Pub/Sub topic.
to_google_cloud_pubsub project_id="my-project", topic_id="alerts", message=textWrites events to one or multiple objects in Google Cloud Storage.
to_google_cloud_storage "gs://my-bucket/data/{uuid}.json" { write_ndjson }Sends unstructured events to a Google SecOps Chronicle instance.
to_google_secops …Sends events as a single HTTP request to a webhook or API endpoint.
to_http "https://example.com/webhook" { write_ndjson }Writes events to one or multiple objects in Amazon S3.
to_s3 "s3://my-bucket/data/{uuid}.json" { write_ndjson }Sends security events to SentinelOne Singularity Data Lake via REST API.
to_sentinelone_data_lake "https://…", …Connects to a remote TCP or TLS endpoint and sends events.
to_tcp "collector.example.com:5044" { write_json }Connects to a remote ZeroMQ subscriber endpoint and sends events.
to_zmq "tcp://collector.example.com:5555", encoding="json", prefix=f"{kind}/"Packages
Section titled “Packages”Parsing
Section titled “Parsing”Parses an incoming bytes stream into events using a string as delimiter.
read_delimited "|"Parses an incoming bytes stream into events using a regular expression as delimiter.
read_delimited_regex r"\s+"Pipelines
Section titled “Pipelines”Printing
Section titled “Printing”Transforms the input event stream to a Newline-Delimited JSON byte stream.
write_ndjson