Connectors

A connector specifies how to exchange raw bytes with an underlying resource, such as a file, a network socket, or a third-party library. A connector provides a loader and/or saver:

1. Loader: translates raw bytes into structured event data
2. Saver: translates structured events into raw bytes

Loaders and savers interact with their corresponding dual from a format:

Connectors appear as an argument to the from and to operators:

from <connector> [read <format>]
to <connector> [write <format>]

If the format is omitted, the default depends on the connector.

Alternatively, instead of a connector, the from and to operators can take a URL or a filesystem path directly:

from <url> [read <format>]
from <path> [read <format>]

to <url> [write <format>]
to <path> [write <format>]

When given a URL, the scheme is used to determine the connector to use. For example, if the URL scheme is http, the http connector is used. The gcs connector is an exception, as it will get used if the URL scheme is gs.

from https://example.com/foo.json
from https https://example.com/foo.json
from https example.com/foo.json

from gs://bucket/logs/log.json
from gcs gs://bucket/logs/log.json

When given a filesystem a path, the file connector is used implicitly. To disambiguate between a relative filesystem path without any slashes or a file extension, and a connector name, the path must contain at least one character that doesn't conform to the pattern [A-Za-z0-9-_]. If the input conforms to that pattern, it's assumed to be a connector name.

# Parsed as a filesystem path
from /tmp/plugin.json
# `plugin` is parsed as a connector name
from plugin
# `./plugin` is parsed as a filesystem path, again (contains a `.` and a `/`)
from ./plugin

Tenzir ships with the following connectors:

📄️ amqp

Sends and receives messages via AMQP.

📄️ azure-blob-storage

Loads from and saves to an Azure Blob Storage

📄️ -

The dash connector (written as -) is an alias for the stdin

📄️ directory

Saves bytes to one file per schema into a directory.

📄️ email

Emails pipeline data through a SMTP server.

📄️ file

Loads bytes from a file. Saves bytes to a file.

📄️ ftp

Loads bytes via FTP.

📄️ ftps

Loads bytes via FTPS.

📄️ gcs

Loads bytes from a Google Cloud Service object. Saves bytes to a Google Cloud

📄️ google-cloud-pubsub

Loads bytes from a Google Cloud Subscription. Publishes bytes to a Google Cloud

📄️ http

Loads and saves bytes via HTTP.

📄️ https

Loads bytes via HTTPS.

📄️ kafka

Loads bytes from and saves bytes to Kafka.

📄️ nic

Reads bytes from a network interface card (NIC).

📄️ s3

Loads from and saves to an Amazon S3 object.

📄️ sqs

Loads bytes from and saves bytes to Amazon SQS queues.

📄️ stdin

The stdin loader is a shorthand for file with argument -.

📄️ stdout

The stdout saver is a shorthand for file with argument -.

📄️ tcp

Loads bytes from a TCP or TLS connection.

📄️ udp

Loads bytes from and saves bytes to a UDP socket.

📄️ zmq

Loads bytes from and saves bytes to ZeroMQ messages.