The tenzir-test
harness discovers and runs
integration tests for pipelines, fixtures, and custom runners. Use this page as
a reference for concepts, configuration, and CLI details. For step-by-step
walkthroughs, see the guides for writing tests,
creating fixtures, and
adding custom runners.
Install
Section titled “Install”tenzir-test
ships as a Python package that requires Python 3.12 or later.
Install it with uv
(or pip
) and verify the
console script:
uv add tenzir-testuvx tenzir-test --help
Core concepts
Section titled “Core concepts”- Project root – Directory passed to
--root
; typically containsfixtures/
,inputs/
,runners/
, andtests/
. - Mode – Auto-detected as project or package. A
package.yaml
in the current directory (or its parent when you run from<package>/tests
) switches to package mode. - Test – Any supported file under
tests/
; frontmatter controls execution. - Runner – Named strategy that executes a test (
tenzir
,python
, custom entries). - Fixture – Reusable environment provider registered under
fixtures/
and requested via frontmatter. - Input – Data accessed with
TENZIR_INPUTS
; defaults to<root>/inputs
but you can override it per directory or per test with aninputs:
setting. - Scratch directory – Ephemeral workspace exposed as
TENZIR_TMP_DIR
during each test run. - Artifact / Baseline – Runner output persisted next to the test; regenerate
with
--update
. - Configuration sources – Frontmatter plus inherited
test.yaml
files;tenzir.yaml
still configures the Tenzir binary.
A typical project layout looks like this:
project-root/├── fixtures/│ └── __init__.py├── inputs/│ └── sample.ndjson├── runners/│ └── __init__.py└── tests/ ├── alerts/ │ ├── sample.tql │ └── sample.txt └── python/ └── quick-check.py
For a package layout (with package.yaml
), the structure may look like:
my-package/├── package.yaml├── operators/│ └── custom-op.tql├── pipelines/│ └── smoke.tql└── tests/ ├── inputs/ │ └── sample.ndjson ├── fixtures/ │ └── __init__.py ├── runners/ │ └── __init__.py └── pipelines/ ├── custom-op.tql └── custom-op.txt
Execution modes and packages
Section titled “Execution modes and packages”- The harness treats
--root
as the project root. If that directory (or its parent when namedtests
) containspackage.yaml
,tenzir-test
switches to package mode and exposes:TENZIR_PACKAGE_ROOT
– Absolute package directory.TENZIR_INPUTS
–<package>/tests/inputs/
unless a directorytest.yaml
or the test frontmatter overrides it.--package-dirs=<package>
– Passed automatically to thetenzir
binary.
- Without a manifest the harness stays in project mode, recursively discovers
tests under
tests/
, and applies global fixtures, runners, and inputs.
CLI reference
Section titled “CLI reference”Run the suite from the project root:
uvx tenzir-test
Useful options:
--tenzir-binary /path/to/tenzir
: Override binary lookup.--tenzir-node-binary /path/to/tenzir-node
: Override node binary path.--update
: Rewrite reference artifacts next to each test.--purge
: Remove generated artifacts (diffs, text outputs) from previous runs.--jobs N
: Control concurrency (4 * CPU cores
by default).--coverage
and--coverage-source-dir
: Enable LLVM coverage.-k
,--keep
: Preserve per-test scratch directories instead of deleting them (same as settingTENZIR_KEEP_TMP_DIRS=1
).--log-comparisons
: Log comparison targets (also viaTENZIR_TEST_LOG_COMPARISONS=1
).--details
: Include runner and fixture metadata next to each test outcome.
Run a subset of tests:
uvx tenzir-test tests/alerts/high-severity.tql
You can list multiple paths in a single invocation. tenzir-test
wires every
argument into the same runner and fixture registry, so you can mix scenarios
from the project and external checkouts:
uvx tenzir-test tests/alerts ../contrib/plugins/*/tests
To regenerate baselines while targeting a specific binary and project root:
TENZIR_BINARY=/opt/tenzir/bin/tenzir \TENZIR_NODE_BINARY=/opt/tenzir/bin/tenzir-node \uvx tenzir-test --root tests --update
Runners
Section titled “Runners”Runner | Command/behaviour | Input extension | Artifact |
---|---|---|---|
tenzir | tenzir -f <test> | .tql | .txt |
python | Execute with the active Python runtime | .py | .txt |
shell | sh -eu <test> via the harness helper | .sh | varies |
Selection flow:
- The harness chooses the first registered runner that claimed the file extension.
- Default suffix mapping applies when no runner explicitly claims an extension:
.tql → tenzir
,.py → python
,.sh → shell
. - A
runner: <name>
frontmatter entry overrides the automatic choice. - If no runner claims the extension and none is specified in frontmatter, the harness fails with an error instead of guessing.
Shell runner
Section titled “Shell runner”Place scripts (for example under tests/shell/
) with the .sh
suffix to run
them under bash -eu
via the shell
runner. The harness also prepends
<root>/_shell
to PATH
so project-specific helper binaries become
discoverable.
Register custom runners in runners/__init__.py
via
tenzir_test.runners.register()
or the @tenzir_test.runners.startup()
decorator. Use replace=True
to override a bundled runner or
register_alias()
to publish alternate names.
The runner guide contains a full example
(XxdRunner
).
Configuration and frontmatter
Section titled “Configuration and frontmatter”tenzir-test
merges configuration sources in this order (later wins):
- Project defaults (
test.yaml
files, applied per directory). - Per-test frontmatter (YAML for
.tql
/.xxd
,# key: value
comments for Python and shell scripts).
Common frontmatter keys:
Key | Type | Default | Description |
---|---|---|---|
runner | string | by suffix | Runner name (tenzir , python , shell , custom). |
fixtures | list of strings | [] | Requested fixtures; use fixture for a single value. |
timeout | integer (s) | 30 | Command timeout. (--coverage multiplies it by five.) |
error | boolean | false | Expect a non-zero exit code. |
skip | string | unset | Mark the test as skipped (reason required). |
inputs | string | project | Override TENZIR_INPUTS for this directory or test. |
test.yaml
files accept the same keys and apply recursively to child
directories. A relative inputs:
value resolves against the file that defines
it, so inputs: ../data
inside tests/alerts/test.yaml
points at
tests/data/
. Frontmatter values follow the same rule and win over directory
defaults. Adjacent tenzir.yaml
files still configure the Tenzir binary; the
harness appends --config=<file>
automatically. The lookup keeps working even
when you point the CLI at extra directories on the command line.
Tenzir configuration files
Section titled “Tenzir configuration files”- The harness inspects the directory that owns each test. If it finds
tenzir.yaml
, it appends--config=<path>
to every invocation of the bundledtenzir
/tql
/diff
runners. The path also seedsTENZIR_CONFIG
unless you set that variable yourself. Custom runners that call the Tenzir binary should either userun.get_test_env_and_config_args(test)
or honour the exported environment variables explicitly. - The built-in
node
fixture uses the same discovery process; see the built-in node fixture section for precedence rules. - This lets you keep one config for CLI-driven scenarios while passing a different config to the embedded node, for example to tweak endpoints or data directories independently.
Fixtures
Section titled “Fixtures”Declaring fixtures
Section titled “Declaring fixtures”- List fixture names in frontmatter (
fixtures: [node, http]
). - The harness encodes requests in
TENZIR_TEST_FIXTURES
and exposes helper APIs intenzir_test.fixtures
:requested()
– Read-only view of active fixtures.require("name")
– Assert that a fixture was requested.Executor()
– Convenience wrapper that runs Tenzir commands with resolved binaries and timeout budget.
Example use from a Python helper:
from tenzir_test.fixtures import Executor
executor = Executor()result = executor.run("from_file 'inputs/events.ndjson' | where severity >= 5\n")assert result.returncode == 0
Built-in node fixture
Section titled “Built-in node fixture”- Request the fixture with
fixtures: [node]
; the harness will starttenzir-node
with the binaries discovered for the current test. - Configuration precedence:
TENZIR_NODE_CONFIG
in the environment.- A
tenzir-node.yaml
placed next to the test file (exported automatically). - The Tenzir defaults (no config file).
- The fixture reuses other inherited arguments (for example
--package-dirs=…
) but replaces any existing--config=
flag so the node process always honours the chosen configuration file. - Tests can read
TENZIR_NODE_CLIENT_ENDPOINT
,TENZIR_NODE_CLIENT_BINARY
, andTENZIR_NODE_CLIENT_TIMEOUT
from the environment to connect to the spawned node. - Pipelines launched by the bundled Tenzir runners automatically receive
--endpoint=<value>
when this fixture is active, so they talk to the transient node without additional wiring. - CLI and node configuration are independent: configure the CLI with
tenzir.yaml
and drop atenzir-node.yaml
(or setTENZIR_NODE_CONFIG
) only when the node needs custom settings.
Registering fixtures
Section titled “Registering fixtures”Implement fixtures in fixtures/
and register them with @tenzir_test.fixture()
.
Decorate a generator function, yield the environment mapping, and handle cleanup
in a finally
block:
from tenzir_test import fixture
@fixture()def http(): server = _start_server() try: yield {"HTTP_FIXTURE_URL": server.url} finally: server.stop()
@fixture
also accepts regular callables returning dictionaries, context
managers, or FixtureHandle
instances for advanced scenarios.
The fixture guide demonstrates an HTTP echo
server that exposes HTTP_FIXTURE_URL
and tears down cleanly.
Environment variables
Section titled “Environment variables”tenzir-test
recognises the following environment variables:
TENZIR_TEST_ROOT
– Default test root when--root
is omitted.TENZIR_BINARY
/TENZIR_NODE_BINARY
– Override binary discovery.TENZIR_INPUTS
– Preferred data directory. Defaults to the project inputs folder but reflects anyinputs:
override fromtest.yaml
or frontmatter.TENZIR_KEEP_TMP_DIRS
– Keep per-test scratch directories (equivalent to--keep
).TENZIR_TEST_LOG_COMPARISONS
– Enable comparison logging.
Fixtures often publish additional variables (for example
TENZIR_NODE_CLIENT_*
, HTTP_FIXTURE_URL
).
During execution the harness also adds transient variables such as
TENZIR_TMP_DIR
so tests and fixtures can create temporary artefacts without
polluting the repository. Combine it with --keep
(or
TENZIR_KEEP_TMP_DIRS=1
) when you need to inspect the generated files after a
run.
Baselines and artifacts
Section titled “Baselines and artifacts”Regenerate reference output whenever behaviour changes intentionally:
uvx tenzir-test --update
--purge
removes stale artifacts (diffs, temporary files). Keep generated
.txt
files under version control so future runs can diff against them.
Troubleshooting
Section titled “Troubleshooting”- Missing binaries – Ensure
tenzir
andtenzir-node
are onPATH
or setTENZIR_BINARY
/TENZIR_NODE_BINARY
explicitly. - Unexpected exits – Set
error: true
in frontmatter when a non-zero exit is expected. - Skipped tests – Use
skip: reason
to document temporary skips; baseline files can stay empty. - Noisy output – Use
--jobs 1
or--log-comparisons
for easier debugging.