s3
Loads from and saves to an Amazon S3 object.
Synopsis
Loader:
s3 [--anonymous] <uri>
Saver:
s3 [--anonymous] <uri>
Description
The s3
loader connects to an S3 bucket to acquire raw bytes from an S3
object. The s3
saver writes bytes to an S3 object in an S3 bucket.
The connector tries to retrieve the appropriate credentials using AWS's default credentials provider chain.
Make sure to configure AWS credentials for the same user account that runs
tenzir
and tenzir-node
. The AWS CLI creates configuration files for the
current user under ~/.aws
, which can only be read by the same user account.
The tenzir-node
systemd unit by default creates a tenzir
user and runs as
that user, meaning that the AWS credentials must also be configured for that
user. The directory ~/.aws
must be readable for the tenzir
user.
If a config file <prefix>/etc/tenzir/plugin/s3.yaml
or
~/.config/tenzir/plugin/s3.yaml
exists, it is always preferred over the
default AWS credentials. The configuration file must have the following format:
<uri>
(Loader, Saver)
The path to the S3 object.
The syntax is
s3://[<access-key>:<secret-key>@]<bucket-name>/<full-path-to-object>(?<options>)
.
Options can be appended to the path as query parameters, as per Arrow:
For S3, the options that can be included in the URI as query parameters are
region
,scheme
,endpoint_override
,allow_bucket_creation
, andallow_bucket_deletion
.
--anonymous
(Loader, Saver)
Ignore any predefined credentials and try to load/save with anonymous credentials.
Examples
Read CSV from an object obj.csv
in the bucket examplebucket
:
from s3 s3://examplebucket/obj.csv read csv
Read JSON from an object test.json
in the bucket examplebucket
, but using a
different, S3-compatible endpoint:
from s3 s3://examplebucket/test.json?endpoint_override=s3.us-west.mycloudservice.com