Packages

This page explains how packages bundle pipelines, operators, contexts, and examples into a deployable unit. You’ll learn about package design principles and how the components fit together.

What is a package

A package is a 1-click deployable unit that implements a specific use case. Packages provide a modular way to distribute TQL functionality, from simple transformations to complete data processing workflows.

Packages solve common distribution challenges:

Reusability: Share operators and pipelines across teams and projects
Configurability: Template inputs let users customize behavior without modifying source files
Testability: The Test Framework validates package behavior
Discoverability: The Tenzir Library makes packages findable and installable

Package structure

A package is a directory with the following structure:

Directorypkg/
- Directorychangelog/ User-facing trail of changes
  - …
- Directoryexamples/ Self-contained snippets to run
  - …
- Directoryoperators/ User-defined operators (UDOs)
  - …
- Directorypipelines/ Fully deployable pipelines
  - …
- Directorytests/ Integration tests
  - Directoryinputs/ Sample data for tests
    …
- package.yaml Package manifest

The package.yaml manifest is the only required file. It identifies the directory as a package and contains metadata, context definitions, and input specifications.

Package components

Operators

User-defined operators (UDOs) in the operators directory provide reusable building blocks. Tenzir names them using the convention <package>::[dirs...]::<basename>.

Operators can accept positional and named arguments for flexible, parameterized transformations.

Pipelines

Pipelines in the pipelines directory are complete, deployable TQL programs. Unlike operators, pipelines must have both input and output operators. They run automatically when you install the package (unless disabled).

Pipeline frontmatter options control restart behavior, enable/disable state, and other runtime settings.

Contexts

Contexts defined in the manifest provide stateful enrichment capabilities. The node creates these contexts when you install the package, making them available for lookup and enrichment operations.

Examples

Examples in the examples directory demonstrate how to use the package. These self-contained snippets appear in the Tenzir Library and help users understand package capabilities.

Tests

Tests in the tests directory validate package behavior using the Test Framework. Tests ensure operators and pipelines work correctly and provide confidence during package updates.

Changelog

The changelog directory tracks user-facing changes using the Ship Framework. It helps users understand what changed between versions and guides upgrade decisions.

Design principles

Composability

Good packages expose operators as building blocks that users can compose in their own pipelines. This maximizes flexibility: users decide when and how to invoke the functionality.

Opt-in execution

Packages that include pipelines execute immediately upon installation. For maximum flexibility, ship pipelines with disabled: true so users can review and enable them explicitly.

Configurability

Use inputs to parameterize packages instead of hardcoding values. This allows the same package to work across different environments without modification.

Testability

Write tests for operators and contexts early. The Test Framework makes testing straightforward, and good test coverage enables confident iteration.

Package lifecycle

Create: Set up the package structure and manifest
Develop: Add operators, pipelines, contexts, and examples
Test: Validate behavior with the Test Framework
Install: Deploy locally or through the Tenzir Library
Maintain: Update the changelog and publish releases