In today's deployment landscape, best practices emphasize GitOps in synergy with
Infrastructure as Code (IaC). With the goal of integrating our services into
these existing mechanisms, we're excited to introduce Pipelines as Code (PaC) in
Tenzir v4.10.
PaC differs from traditional deployment methods in two key aspects. Firstly,
pipelines deployed as code always start with the Tenzir node, ensuring
continuous operation. Secondly, to safeguard them, deletion via the user
interface is disallowed for pipelines deployed as code.
Did you ever try to run Tenzir in Docker on a new-ish MacBook and encountered
this error?
Now, this works as expected:
This works because the Tenzir Docker images now are multi-archecture images
built natively for both linux/amd64 and linux/arm64/v8. In addition to
supporting M-series MacBooks, this also allows the Docker images to run without
emulation on other arm64-based systems like AWS Graviton.
We've substituted the tenzir.allow-unsafe-pipelines feature with
tenzir.no-location-overrides, flipping the default set-up and enhancing user
experience.
tenzir.allow-unsafe-pipelines had been historically puzzling for newcomers
given its seemingly fearsome name and ambiguous implications. Why would someone
consciously permit unsafe pipelines? And why have we now defaulted to allowing
them?
Pipelines have the ability to execute in multiple processes. For instance,
executing tenzir 'from file.json | import' would prompt from file.json to
run in the tenzir process, and import in the connected tenzir-node
process. An operator's location can be assigned as local, anywhere, or remote.
On initializing a pipeline, Tenzir's executor intelligently divides the pipeline
according to location change between local and remote, starts separated
pipelines at their respective locations, connects them to one another.
However, operator locations can also be manually manipulated. For instance, when
capturing PCAPs, users might desire to prevent unnecessary inter-process
communication and directly connect the Tenzir Node to the network
interface—achieved by executing tenzir 'remote from nic …'. This command
instructs the executor to consistently run from nic … directly at the node.
When introducing this feature during the Tenzir v4.0 release, we wanted to be
cautious about unrestricted use of this feature, leading to the creation of the
tenzir.allow-unsafe-pipelines option, which by default was set to false. This
option prohibits the use of location overrides when enabled but simultaneously
posed puzzlement to new users being the lone feature disallowed in an "unsafe"
pipeline.
In response to feedback, we've improved our approach. Location overrides are now
permitted by default and can be disallowed by using the new option
tenzir.no-location-overrides.
Did you ever want to act on multiple fields in enrich or lookup? Now you
can!
For example, you can now use a GeoIP context on all IP
addresses in your data as simple as this:
You can also specify multiple fields explicitly:
… | enrich country --field src_ip,dest_ip
The output of lookup and enrich changed slightly to accomodate multiple
contexts in the same event. Under the output field (that defaults to the context
name), there is now a new record named context, under which we replicate the
path to the enriched fields for placing the context. That is, the context of
id.orig_h in this example is accessible as country.context.id.orig_h: