New S3 Operator & Fork Operator Bugfix

This release adds a new S3 operator and fixes a bug within the fork operator.

Download the release on GitHub.

Features

from_s3 operator

The from_s3 operator reads files from Amazon S3 with support for glob patterns, automatic format detection, and file monitoring.

from_s3 "s3://my-bucket/data/**.json"

The operator supports multiple authentication methods including default AWS credentials, explicit access keys, IAM role assumption, and anonymous access for public buckets:

from_s3 "s3://my-bucket/data.csv",
  access_key=secret("AWS_ACCESS_KEY"),
  secret_key=secret("AWS_SECRET_KEY")

For S3-compatible services, specify custom endpoints via URL parameters:

from_s3 "s3://my-bucket/data/**.json?endpoint_override=minio.example.com:9000&scheme=http"

Additional features include file watching for continuous ingestion, automatic file removal or renaming after processing, and path field injection to track source files in events.

By @raxyte in #5449.

Bug Fixes

Fix fork operator stopping after initial events

We fixed a bug where the fork operator would stop processing events after handling only the first few events, causing data loss in downstream pipeline stages.

By @raxyte in #5450.