This release adds a new S3 operator and fixes a bug within the fork operator.
🚀 Features
Section titled “🚀 Features”from_s3 operator
Section titled “from_s3 operator”Aug 28, 2025 · @raxyte · #5449
The from_s3 operator reads files from Amazon S3 with support for glob
patterns, automatic format detection, and file monitoring.
from_s3 "s3://my-bucket/data/**.json"The operator supports multiple authentication methods including default AWS credentials, explicit access keys, IAM role assumption, and anonymous access for public buckets:
from_s3 "s3://my-bucket/data.csv", access_key=secret("AWS_ACCESS_KEY"), secret_key=secret("AWS_SECRET_KEY")For S3-compatible services, specify custom endpoints via URL parameters:
from_s3 "s3://my-bucket/data/**.json?endpoint_override=minio.example.com:9000&scheme=http"Additional features include file watching for continuous ingestion, automatic file removal or renaming after processing, and path field injection to track source files in events.
🐞 Bug Fixes
Section titled “🐞 Bug Fixes”Fix fork operator stopping after initial events
Section titled “Fix fork operator stopping after initial events”We fixed a bug where the fork operator would stop processing events after handling
only the first few events, causing data loss in downstream pipeline stages.