🚀 Features
Section titled “🚀 Features”Add SQS Connector
Section titled “Add SQS Connector”The new sqs connector makes it possible to read from and write to Amazon SQS
queues.
Add the set operator for upserting fields
Section titled “Add the set operator for upserting fields”Mar 21, 2024 · @dominiklohmann · #4057
The new set operator upserts fields, i.e., acts like replace for existing
fields and like extend for new fields. It also supports setting the schema
name explicitly via set #schema="new-name".
The put operator now supports setting the schema name explicitly via put #schema="new-name".
Add an email saver
Section titled “Add an email saver”The new email saver allows for sending pipeline data via email by connecting
to a mail server via SMTP or SMTPS.
Introduce the every operator modifier
Section titled “Introduce the every operator modifier”Mar 19, 2024 · @dominiklohmann · #4050
The every <interval> operator modifier executes a source operator repeatedly.
For example, every 1h from http://foo.com/bar polls an endpoint every hour.
Introduce --replace, --separate, and --yield for contexts
Section titled “Introduce --replace, --separate, and --yield for contexts”Mar 15, 2024 · @dominiklohmann · #4040
The --replace option for the enrich operator causes the input values to be
replaced with their context instead of extending the event with the context,
resulting in a leaner output.
The --separate option makes the enrich and lookup operators handle each
field individually, duplicating the event for each relevant field, and
returning at most one context per output event.
The --yield <field> option allows for adding only a part of a context with the
enrich and lookup operators. For example, with a geoip context with a
MaxMind country database, --yield registered_country.iso_code will cause the
enrichment to only consist of the country’s ISO code.
Add a files source
Section titled “Add a files source”Mar 15, 2024 · @dominiklohmann · #4035
The new files source lists file information for a given directory.
🔧 Changes
Section titled “🔧 Changes”Introduce --replace, --separate, and --yield for contexts
Section titled “Introduce --replace, --separate, and --yield for contexts”Mar 15, 2024 · @dominiklohmann · #4040
The enrich and lookup operators now include the metadata in every context
object to accomodate the new --replace and --separate options. Previously,
the metadata was available once in the output field.
The mode field in the enrichments returned from the lookup operator is now
lookup.retro, lookup.live, or lookup.snapshot depending on the mode.
The bloom-filter context now always returns true or null for the context
instead of embedding the result in a record with a single data field.
🐞 Bug Fixes
Section titled “🐞 Bug Fixes”Add SQS Connector
Section titled “Add SQS Connector”Source operators that do not quit on their own only freed their resources after they had emitted an additional output, even after the pipeline had already exited. This sometimes caused errors when restarting pipelines, and in rare cases caused Tenzir nodes to hang on shutdown. This no longer happens, and the entire pipeline shuts down at once.
Fix disk metrics with custom state directories
Section titled “Fix disk metrics with custom state directories”Mar 21, 2024 · @dominiklohmann · #4058
Disk metrics now work correctly for deployments with a customized state directory.
Fix invalid assertion in compress operator
Section titled “Fix invalid assertion in compress operator”Mar 18, 2024 · @dominiklohmann · #4048
The compress and to operators no longer fail when compression is unable to
further reduce the size of a batch of bytes.
Fix an off-by-one error when loading persisted contexts
Section titled “Fix an off-by-one error when loading persisted contexts”Mar 17, 2024 · @dominiklohmann · #4045
We fixed a bug that caused every second context to become unavailable after a restarting the node.
Introduce --replace, --separate, and --yield for contexts
Section titled “Introduce --replace, --separate, and --yield for contexts”Mar 15, 2024 · @dominiklohmann · #4040
drop and select silently ignored all but the first match of the specified
type extractors and concepts. This no longer happens. For example, drop :time
drops all fields with type time from events.
Enriching a field in adjacent events in lookup and enrich with a
lookup-table context sometimes crashed when the lookup-table referred to
values of different types.
The geoip context sometimes returned incorrect values. This no longer happens.
Fix from <url> with username
Section titled “Fix from <url> with username”Mar 15, 2024 · @jachris · #4043
from <url> now also works when the url specifies username and password.