Glossary
This page defines central terms in the Tenzir ecosystem.
If you are missing a term, please open a GitHub Discussion or ping us in our Discord chat.
App
Web user interface to access platform at app.tenzir.com.
The app is a web application that partially runs in the user's browser. It is written in Svelte.
Catalog
Maintains partition ownership and metadata.
The catalog is a component in the node that owns the partitions, keeps metadata about them, and maintains a set of sparse secondary indexes to identify relevant partitions for a given query. It offers a transactional interface for adding and removing partitions.
Connector
Manages chunks of raw bytes by interacting with a resource.
A connector is either a loader that acquires bytes from a resource, or a
saver that sends bytes to a resource. Example connectors are
file
, kafka
, and
nic
.
- See all available connectors
Context
A stateful object used for in-band enrichment.
Contexts live inside a node and you can manage them with the
context
operator. A context has pluggable type, such
as a lookup table, GeoIP database, or a custom plugin. The
enrich
places a context into a pipeline for
enrichment.
Format
Translates between bytes and events.
A format is either a parser that converts bytes to events, or a printer
that converts events to bytes. Example formats are json
,
cef
, and pcap
.
- See all available formats
Index
Optional data structures for accelerating historical queries.
Tenzir has sparse indexes. Sparse indexes live in memory and point to partitions.
Integration
A set of pipelines to integrate with a third-party product.
An integration describes use cases in combination with a specific product or tool. Based on the depth of the configuration, this may require configuration on either end.
Loader
A connector that acquires bytes.
A loader is the dual to a saver. It has a no input and only performs a
side effect that acquires bytes. Use a loader in the
from
or load
operators.
- Learn more about pipelines
- See all connectors
Node
A host for pipelines and storage reachable over the network.
The tenzir-node
binary starts a node in a dedicated server process that
listens on TCP port 5158.
- Deploy a node
- Use the REST API to manage a node
- Import into a node
- Export from a node
Metrics
Runtime statistics about pipeline execution.
Operator
The building block of a pipeline.
An operator is a source, transformation, or sink.
- See all available operators
PaC
The acronym PaC stands for Pipelines as Code. It is meant as an adaptation of Infrastructure as Code (IaC) with pipelines represent the (data) infrastructure that is provisioning as code.
- Learn how to provision piplines as code.
Parser
A format that translates bytes into events.
A parser is the dual to a printer. Use a parser in the
from
or read
operators. You
can use the parse
operator to parse a single field
with a parser.
- Learn more about pipelines
- See all formats
Partition
The horizontal scaling unit of the storage attached to a node.
A partition contains the raw data and optionally a set of indexes. Supported formats are Parquet or Feather.
- Control the partition size
- Configure catalog and partition indexes
- Select the store format
- Adjust the store compression
- Rebuild partitions
Pipeline
Combines a set of operators into a dataflow graph.
- Understand how pipelines work
- Understand the pipeline language
- Run a pipeline
Platform
Control plane for nodes and pipelines, accessible through app at app.tenzir.com.
Printer
A format that translates events into bytes.
A printer is the dual to a parser. Use a parser in the
to
or write
operators.
- Learn more about pipelines
- See all formats
Saver
A connector that emits bytes.
A saver is the dual to a loader. It has a no output and only performs
a side effect that emits bytes. Use a saver in the to
or
save
operators.
- Learn more about pipelines
- See all connectors
Schema
A named record type describing the top-level structure of a data frame.
Sink
An operator consuming input, without producing any output.
- Learn more about pipelines
Source
An operator producing output, without consuming any input.
- Learn more about pipelines
TQL
An acronym for Tenzir Query Language.
TQL is the language in which users write pipelines.
Transformation
An operator consuming both input and producing output.
- Learn more about pipelines