Skip to main content
Version: Next

Run pipelines

You can run a pipeline in the app or on the command line using the tenzir binary.

In the app

Run a pipeline by writing typing it in the editor and hitting the Run button.

The following invariants apply:

  1. You must start with a source
  2. The browser is always the sink

The diagram below illustrates these mechanics:

For example, write version and click Run to see a single event arrive.

On the command line

On the command line, run tenzir <pipeline> where <pipeline> is the definition of the pipeline.

If the pipeline expects events as its input, an implicit load - | read json will be prepended. If it expects bytes instead, only load - is prepended. Likewise, if the pipeline outputs events, an implicit write json | save - will be appended. If it outputs bytes instead, only save - is appended.

The diagram below illustrates these mechanics:

For example, run tenzir 'version' to see a single event rendered as JSON:

{
"version": "v4.6.4-155-g0b75e93026",
"major": 4,
"minor": 6,
"patch": 4,
"tweak": 155
}

You could also render it differently by passing a different format to write, or by inferring the format from the file extension:

tenzir 'version | write csv'
tenzir 'version | to /tmp/version.ssv'
tenzir 'version | to /tmp/version.parquet'

Instead of passing the pipeline description to the tenzir executable, you can also load the definition from a file via -f:

tenzir -f pipeline.tql

This will interpret the file contents as pipeline and run it.

As Code

In addition to running pipelines interactively, you can also deploy pipelines as code (PaC). This infrastructure-as-code-like method differs from the app-based deployment in two ways:

  1. Pipelines deployed as code always start with the Tenzir node, ensuring continuous operation.
  2. To safeguard them, deletion via the user interface is disallowed.

Here's a an example of deploying a pipeline through your configuration:

<prefix>/etc/tenzir/tenzir.yaml
tenzir:
pipelines:
# A unique identifier for the pipeline that's used for metrics, diagnostics,
# and API calls interacting with the pipeline.
suricata-over-tcp:
# An optional user-facing name for the pipeline. Defaults to the id.
name: Import Suricata from TCP
# The definition of the pipeline. Configured pipelines that fail to start
# cause the node to fail to start.
definition: |
from tcp://0.0.0.0:34343 read suricata
| import
# Pipelines that encounter an error stop running and show an error state.
# This option causes pipelines to automatically restart when they
# encounter an error instead. The first restart happens immediately, and
# subsequent restarts after the configured delay, defaulting to 1 minute.
# The following values are valid for this option:
# - Omit the option, or set it to null or false to disable.
# - Set the option to true to enable with the default delay of 1 minute.
# - Set the option to a valid duration to enable with a custom delay.
restart-on-error: 1 minute
# Add a list of labels that are shown in the pipeline overview page at
# app.tenzir.com.
labels:
- Suricata
- Import