Tenzir Node v4.30: ClickHouse Integration

Tenzir Node v4.30 introduces a to_clickhouse operator and streamlines the TLS settings for all operators.

ClickHouse Integration

Tenzir can now seamlessly write data to a ClickHouse table, using the new to_clickhouse operator. The operator uses ClickHouse's C++ client library clickhouse-cpp to efficiently insert blocks in a columnar fashion. It supports all of Tenzir's types and can both create tables as well as append to existing ones:

from { i: 42, d: 10.0, b: true, l: [42], r:{ s:"string" } }
to_clickhouse table="example", primary=i

Streamlined TLS Settings

Many operators feature some TLS settings, that all worked very similar, but not identical. With v4.30, the TLS settings are now are now identical between all operators and all share the same semantics.

Additionally, there is a new configuration option tenzir.cacert, which allows setting of a global default for the CA certificates file. The default value will be chosen appropriately for the system the node runs on. Of course the per-operator arguments still take precedence.

Set a cacert default in the config

# /opt/tenzir/etc/tenzir/tenzir.yaml
tenzir:
  cacert: path/to/custom.bundle

TLS by default

With these changes we also enabled TLS by default on most operators. This may cause some pipelines that rely on TLS not being enabled to stop working. If you want to explicitly disable TLS, you can still do so by providing tls=false to these operators.

Pipeline Metrics

Tenzir previously only featured very detailed operator metrics, which provided a whole lot of detail, but were both cumbersome to work with and fairly expensive to collect.

This release introduces pipeline metrics, which give you information about the throughput of entire pipelines. These metrics are much easier to work with, for example to create Dashboards:

Ingress of all Pipelines

metrics "pipeline"
chart_line x=timestamp, y={
  "Ingress Gb/s": mean(ingress.bytes / 1G / ingress.duration.count_seconds()),
}, resolution=10s, group=pipeline_id, x_min = now()-1d

Deprecation Notice

The old Operator Metrics are now deprecated and will be removed in some future release.

Fixes & Updates

This release also contains a few small fixes. Most notably we enabled native Google Cloud Storage support to TQL2 in the form of the load_gcs and save_gcs operators. These are also supported using the gs:// URI scheme in the generic from and to operators.

Let's Connect!

Join our Discord server, to ask questions, discuss features or just hang out! Here we also host our bi-weekly office hours (every second Tuesday at 5 PM CET), where we showcase the latest features, give sneak peaks into upcoming ones, answer your questions or just discuss general tech topics! We are looking forward to getting in touch with you!