Skip to content

Test Framework

The tenzir-test harness discovers and runs integration tests for pipelines, fixtures, and custom runners. Use this page as a reference for concepts, configuration, and CLI details. For step-by-step walkthroughs, see the guides for writing tests, creating fixtures, and adding custom runners.

tenzir-test ships as a Python package that requires Python 3.12 or later. Install it with uv (or pip) and verify the console script:

Terminal window
uv add tenzir-test
uvx tenzir-test --help
  • Project root – Directory passed to --root; typically contains fixtures/, inputs/, runners/, and tests/.
  • Mode – Auto-detected as project or package. A package.yaml in the current directory (or its parent when you run from <package>/tests) switches to package mode.
  • Test – Any supported file under tests/; frontmatter controls execution.
  • Runner – Named strategy that executes a test (tenzir, python, custom entries).
  • Fixture – Reusable environment provider registered under fixtures/ and requested via frontmatter.
  • Input – Data accessed with TENZIR_INPUTS; defaults to <root>/inputs but you can override it per directory or per test with an inputs: setting.
  • Scratch directory – Ephemeral workspace exposed as TENZIR_TMP_DIR during each test run.
  • Artifact / Baseline – Runner output persisted next to the test; regenerate with --update.
  • Configuration sources – Frontmatter plus inherited test.yaml files; tenzir.yaml still configures the Tenzir binary.

A typical project layout looks like this:

project-root/
├── fixtures/
│ └── __init__.py
├── inputs/
│ └── sample.ndjson
├── runners/
│ └── __init__.py
└── tests/
├── alerts/
│ ├── sample.tql
│ └── sample.txt
└── python/
└── quick-check.py

For a package layout (with package.yaml), the structure may look like:

my-package/
├── package.yaml
├── operators/
│ └── custom-op.tql
├── pipelines/
│ └── smoke.tql
└── tests/
├── inputs/
│ └── sample.ndjson
├── fixtures/
│ └── __init__.py
├── runners/
│ └── __init__.py
└── pipelines/
├── custom-op.tql
└── custom-op.txt
  • The harness treats --root as the project root. If that directory (or its parent when named tests) contains package.yaml, tenzir-test switches to package mode and exposes:
    • TENZIR_PACKAGE_ROOT – Absolute package directory.
    • TENZIR_INPUTS<package>/tests/inputs/ unless a directory test.yaml or the test frontmatter overrides it.
    • --package-dirs=<package> – Passed automatically to the tenzir binary.
  • Without a manifest the harness stays in project mode, recursively discovers tests under tests/, and applies global fixtures, runners, and inputs.

Run the suite from the project root:

Terminal window
uvx tenzir-test

Useful options:

  • --tenzir-binary /path/to/tenzir: Override binary lookup.
  • --tenzir-node-binary /path/to/tenzir-node: Override node binary path.
  • --update: Rewrite reference artifacts next to each test.
  • --purge: Remove generated artifacts (diffs, text outputs) from previous runs.
  • --jobs N: Control concurrency (4 * CPU cores by default).
  • --coverage and --coverage-source-dir: Enable LLVM coverage.
  • -k, --keep: Preserve per-test scratch directories instead of deleting them (same as setting TENZIR_KEEP_TMP_DIRS=1).
  • --log-comparisons: Log comparison targets (also via TENZIR_TEST_LOG_COMPARISONS=1).
  • --details: Include runner and fixture metadata next to each test outcome.

Run a subset of tests:

Terminal window
uvx tenzir-test tests/alerts/high-severity.tql

You can list multiple paths in a single invocation. tenzir-test wires every argument into the same runner and fixture registry, so you can mix scenarios from the project and external checkouts:

Terminal window
uvx tenzir-test tests/alerts ../contrib/plugins/*/tests

To regenerate baselines while targeting a specific binary and project root:

Terminal window
TENZIR_BINARY=/opt/tenzir/bin/tenzir \
TENZIR_NODE_BINARY=/opt/tenzir/bin/tenzir-node \
uvx tenzir-test --root tests --update
RunnerCommand/behaviourInput extensionArtifact
tenzirtenzir -f <test>.tql.txt
pythonExecute with the active Python runtime.py.txt
shellsh -eu <test> via the harness helper.shvaries

Selection flow:

  1. The harness chooses the first registered runner that claimed the file extension.
  2. Default suffix mapping applies when no runner explicitly claims an extension: .tql → tenzir, .py → python, .sh → shell.
  3. A runner: <name> frontmatter entry overrides the automatic choice.
  4. If no runner claims the extension and none is specified in frontmatter, the harness fails with an error instead of guessing.

Place scripts (for example under tests/shell/) with the .sh suffix to run them under bash -eu via the shell runner. The harness also prepends <root>/_shell to PATH so project-specific helper binaries become discoverable.

Register custom runners in runners/__init__.py via tenzir_test.runners.register() or the @tenzir_test.runners.startup() decorator. Use replace=True to override a bundled runner or register_alias() to publish alternate names.

The runner guide contains a full example (XxdRunner).

tenzir-test merges configuration sources in this order (later wins):

  1. Project defaults (test.yaml files, applied per directory).
  2. Per-test frontmatter (YAML for .tql/.xxd, # key: value comments for Python and shell scripts).

Common frontmatter keys:

KeyTypeDefaultDescription
runnerstringby suffixRunner name (tenzir, python, shell, custom).
fixtureslist of strings[]Requested fixtures; use fixture for a single value.
timeoutinteger (s)30Command timeout. (--coverage multiplies it by five.)
errorbooleanfalseExpect a non-zero exit code.
skipstringunsetMark the test as skipped (reason required).
inputsstringprojectOverride TENZIR_INPUTS for this directory or test.

test.yaml files accept the same keys and apply recursively to child directories. A relative inputs: value resolves against the file that defines it, so inputs: ../data inside tests/alerts/test.yaml points at tests/data/. Frontmatter values follow the same rule and win over directory defaults. Adjacent tenzir.yaml files still configure the Tenzir binary; the harness appends --config=<file> automatically. The lookup keeps working even when you point the CLI at extra directories on the command line.

  • The harness inspects the directory that owns each test. If it finds tenzir.yaml, it appends --config=<path> to every invocation of the bundled tenzir/tql/diff runners. The path also seeds TENZIR_CONFIG unless you set that variable yourself. Custom runners that call the Tenzir binary should either use run.get_test_env_and_config_args(test) or honour the exported environment variables explicitly.
  • The built-in node fixture uses the same discovery process; see the built-in node fixture section for precedence rules.
  • This lets you keep one config for CLI-driven scenarios while passing a different config to the embedded node, for example to tweak endpoints or data directories independently.
  • List fixture names in frontmatter (fixtures: [node, http]).
  • The harness encodes requests in TENZIR_TEST_FIXTURES and exposes helper APIs in tenzir_test.fixtures:
    • requested() – Read-only view of active fixtures.
    • require("name") – Assert that a fixture was requested.
    • Executor() – Convenience wrapper that runs Tenzir commands with resolved binaries and timeout budget.

Example use from a Python helper:

from tenzir_test.fixtures import Executor
executor = Executor()
result = executor.run("from_file 'inputs/events.ndjson' | where severity >= 5\n")
assert result.returncode == 0
  • Request the fixture with fixtures: [node]; the harness will start tenzir-node with the binaries discovered for the current test.
  • Configuration precedence:
    1. TENZIR_NODE_CONFIG in the environment.
    2. A tenzir-node.yaml placed next to the test file (exported automatically).
    3. The Tenzir defaults (no config file).
  • The fixture reuses other inherited arguments (for example --package-dirs=…) but replaces any existing --config= flag so the node process always honours the chosen configuration file.
  • Tests can read TENZIR_NODE_CLIENT_ENDPOINT, TENZIR_NODE_CLIENT_BINARY, and TENZIR_NODE_CLIENT_TIMEOUT from the environment to connect to the spawned node.
  • Pipelines launched by the bundled Tenzir runners automatically receive --endpoint=<value> when this fixture is active, so they talk to the transient node without additional wiring.
  • CLI and node configuration are independent: configure the CLI with tenzir.yaml and drop a tenzir-node.yaml (or set TENZIR_NODE_CONFIG) only when the node needs custom settings.

Implement fixtures in fixtures/ and register them with @tenzir_test.fixture(). Decorate a generator function, yield the environment mapping, and handle cleanup in a finally block:

from tenzir_test import fixture
@fixture()
def http():
server = _start_server()
try:
yield {"HTTP_FIXTURE_URL": server.url}
finally:
server.stop()

@fixture also accepts regular callables returning dictionaries, context managers, or FixtureHandle instances for advanced scenarios.

The fixture guide demonstrates an HTTP echo server that exposes HTTP_FIXTURE_URL and tears down cleanly.

tenzir-test recognises the following environment variables:

  • TENZIR_TEST_ROOT – Default test root when --root is omitted.
  • TENZIR_BINARY / TENZIR_NODE_BINARY – Override binary discovery.
  • TENZIR_INPUTS – Preferred data directory. Defaults to the project inputs folder but reflects any inputs: override from test.yaml or frontmatter.
  • TENZIR_KEEP_TMP_DIRS – Keep per-test scratch directories (equivalent to --keep).
  • TENZIR_TEST_LOG_COMPARISONS – Enable comparison logging.

Fixtures often publish additional variables (for example TENZIR_NODE_CLIENT_*, HTTP_FIXTURE_URL).

During execution the harness also adds transient variables such as TENZIR_TMP_DIR so tests and fixtures can create temporary artefacts without polluting the repository. Combine it with --keep (or TENZIR_KEEP_TMP_DIRS=1) when you need to inspect the generated files after a run.

Regenerate reference output whenever behaviour changes intentionally:

Terminal window
uvx tenzir-test --update

--purge removes stale artifacts (diffs, temporary files). Keep generated .txt files under version control so future runs can diff against them.

  • Missing binaries – Ensure tenzir and tenzir-node are on PATH or set TENZIR_BINARY / TENZIR_NODE_BINARY explicitly.
  • Unexpected exits – Set error: true in frontmatter when a non-zero exit is expected.
  • Skipped tests – Use skip: reason to document temporary skips; baseline files can stay empty.
  • Noisy output – Use --jobs 1 or --log-comparisons for easier debugging.

Last updated: