🚀 Features
Section titled “🚀 Features”Add a package manager
Section titled “Add a package manager”The new package operator allows for adding and removing packages, a
combination of pipelines and contexts deployed to a node as a set. Nodes load
packages installed to <configdir>/tenzir/package/<package-name>/package.yaml
on startup.
Add the buffer operator for breaking back pressure
Section titled “Add the buffer operator for breaking back pressure”Jul 19, 2024 · @dominiklohmann · #4404
The buffer operator buffers up to the specified number of events in an
in-memory buffer. By default, operators in a pipeline run only when their
downstream operators want to receive input. This mechanism is called back
pressure. The buffer operator effectively breaks back pressure by storing up
to the specified number of events in memory, always requesting more input, which
allows upstream operators to run uninterruptedly even in case the downstream
operators of the buffer are unable to keep up. This allows pipelines to handle
data spikes more easily.
🔧 Changes
Section titled “🔧 Changes”Refactor python operator setup
Section titled “Refactor python operator setup”The python operator now resolves dependencies with every fresh pipeline run.
Just restart your pipeline to upgrade to the latest available versions of your
Python modules.
The python operator no longer uses pip but rather
uv. In case you set custom environment
variables for pip you need to exchange those with alternative settings that
work with uv.
Remove the use_simple_format option for /serve
Section titled “Remove the use_simple_format option for /serve”Jul 22, 2024 · @dominiklohmann · #4411
The /serve endpoint now always uses the simple output format for schema
definitions. The option use_simple_format is now ignored.
🐞 Bug Fixes
Section titled “🐞 Bug Fixes”Add the buffer operator for breaking back pressure
Section titled “Add the buffer operator for breaking back pressure”Jul 19, 2024 · @dominiklohmann · #4404
Metrics emitted towards the end of an operator’s runtime were sometimes not recorded correctly. This now works reliably.