Overview

This guide provides an overview of data collection in TQL. You’ll learn about the different approaches for ingesting data from various sources.

TQL provides two types of input operators:

from_* operators like from_file and from_http read bytes and parse them using a subpipeline.
Direct event operators like from_kafka and from_udp produce structured events directly without an intermediate byte stream.

Collection patterns

Different data sources require different collection approaches.

Files and cloud storage

Read local files, watch directories for changes, or access cloud storage:

// Single file with automatic format detection
from_file "/var/log/app.json"

// Watch a directory for new files
from_file "/incoming/*.csv", watch=true

// Cloud storage with glob patterns
from_file "s3://bucket/data/**/*.parquet"

See the file reading guide for details.

HTTP and APIs

Fetch data from web APIs with authentication, pagination, and retry handling:

from_http "https://api.example.com/events",
  headers={"Authorization": "Bearer " + secret("API_TOKEN")}

See the HTTP and API guide for pagination patterns and advanced configurations.

Message brokers

Subscribe to topics or queues from Apache Kafka, AMQP, Amazon SQS, and Google Cloud Pub/Sub:

from_kafka "security-events", offset="end"

See the message broker guide for broker-specific configurations.

Network data

Receive data over TCP or UDP sockets, or capture packets from network interfaces:

// UDP syslog receiver
from_udp "0.0.0.0:514"

// TCP with TLS
from "tcp://0.0.0.0:8443", tls=true

See the network data guide for socket configurations and packet capture.

Sending data to destinations

For routing data to outputs, see the Routing guides, which cover destination operators, file output, load balancing, and pipeline connections.