This release brings forth stability improvements under high load that could cause platform unresponsiveness, fixes API request isolation problems, better kafka diagnostics and more.
Download the release on GitHub.
Features
Section titled “Features”Subscribe to multiple topics at once
Section titled “Subscribe to multiple topics at once”The subscribe operator now accepts multiple topics to subscribe to. For
example, subscribe "notices", "alerts" subscribes to both the notices, and
the alerts topic. This makes it easier to build pipelines that join multiple
topics back together.
from_gcs operator
Section titled “from_gcs operator”The new from_gcs operator reads files from Google Cloud Storage with support
for glob patterns, authentication via Application Default Credentials, and all
standard file processing features like monitoring and path tracking.
Changes
Section titled “Changes”Better kafka diagnostics
Section titled “Better kafka diagnostics”The kafka related operators now emit more diagnostics in unexpected
situations, providing more information to diagnose problems.
Better defaults for load_kafka
Section titled “Better defaults for load_kafka”The load_kafka operators previously used offset="end" as the default, which
meant that it always started from the end of the topic. This default was now
changed to "stored", such that the previously commited offset is used instead.
Bug Fixes
Section titled “Bug Fixes”Pipeline execution under high loads
Section titled “Pipeline execution under high loads”Previously, the execution of certain pipelines under high load scenarios could lead to general unresponsiveness. In extreme cases, this meant that the platform wasn’t able to reach the node. This issue has now been resolved, leading to a more reliable and responsive experience.
API request isolation
Section titled “API request isolation”Requests to the /pipeline API are now properly isolated and sequentialized.
Before, it could happen that certain requests that should not be executed
concurrently were interleaved. This could lead to unpredictable results when
interacting with pipelines through the platform.
session_name and external_id in aws_iam options
Section titled “session_name and external_id in aws_iam options”The load_kafka, save_kafka and to_kafka operators now accept configuring
session_name and external_id for aws_iam options.
Behavior of the throttle operator
Section titled “Behavior of the throttle operator”The throttle operator now correctly forwards its data in a timely manner.
Hang in every and cron
Section titled “Hang in every and cron”We fixed a bug in every and cron operators that could cause them to hang
and panic with assertions failures.