Glossary

This page defines central terms in the Tenzir ecosystem.

missing term?

If you are missing a term, please open a GitHub Discussion or ping us in our Discord chat.

App

Web user interface to access platform at app.tenzir.com.

The app is a web application that partially runs in the user's browser. It is written in Svelte.

• Use the app

Catalog

Maintains partition ownership and metadata.

The catalog is a component in the node that owns the partitions, keeps metadata about them, and maintains a set of sparse secondary indexes to identify relevant partitions for a given query. It offers a transactional interface for adding and removing partitions.

• Tune catalog fragmentation
• Configure catalog and partition indexes

Connector

Manages chunks of raw bytes by interacting with a resource.

A connector is either a loader that acquires bytes from a resource, or a saver that sends bytes to a resource. Example connectors are file, kafka, and nic.

• See all available connectors

Format

Translates between bytes and events.

A format is either a parser that converts bytes to events, or a printer that converts events to bytes. Example formats are json, cef, and pcap.

• See all available formats

Index

Optional data structures for accelerating historical queries.

Tenzir has sparse and dense indexes. Sparse indexes live in memory and point to partitions, whereas dense indexes live within a partition and point to individual rows within the partition.

• Configure catalog and partition indexes

Node

A host for pipelines and storage reachable over the network.

The tenzir-node binary starts a node in a dedicated server process that listens on TCP port 5158.

• Deploy a node
• Use the app to manage a node
• Use the REST API to manage a node
• Import into a node
• Export from a node

Metrics

Runtime statistics about pipeline execution.

• Collect metrics

Operator

The building block of a pipeline.

An operator is a source, transformation, or sink.

• See all available operators

Partition

The horizontal scaling unit of the storage attached to a node.

A partition contains the raw data and optionally a set of indexes. Supported formats are Parquet or Feather.

• Control the partition size
• Configure catalog and partition indexes
• Select the store format
• Adjust the store compression
• Rebuild partitions

Pipeline

Combines a set of operators into a dataflow graph.

• Understand the pipeline language
• Run a pipeline

Platform

Control plane for nodes and pipelines, accessible through app at app.tenzir.com.

• Use the app

Schema

A named record type describing the top-level structure of a data frame.

Schemas

• Show available schemas

Sink

An operator consuming input, without producing any output.

• See all available sinks

Source

An operator producing output, without consuming any input.

• See all available sources

TQL

An acronym for Tenzir Query Language.

TQL is the language in which users write pipelines.

• Learn more about the language

Transformation

An operator consuming both input and producing output.

• See all transformations