Get data from the network

This guide shows you how to receive data directly from network sources using TQL. You’ll learn to listen on TCP and UDP sockets for incoming data and capture raw packets from network interfaces.

TCP sockets

The Transmission Control Protocol (TCP) provides reliable, ordered byte streams. Use TCP when you need guaranteed delivery and message ordering.

Listen for connections

Start a TCP server that accepts incoming connections:

from "tcp://0.0.0.0:9000" {
  read_json
}

This listens on all interfaces (0.0.0.0) on port 9000. Specify a parsing pipeline to convert incoming bytes to events.

Connect to a remote server

Act as a TCP client by connecting to an existing server:

from "tcp://192.168.1.100:9000", connect=true {
  read_json
}

Enable TLS

Secure your TCP connections with TLS by passing a tls record:

from "tcp://0.0.0.0:9443", tls={certfile: "cert.pem", keyfile: "key.pem"} {
  read_json
}

For testing, generate certificates with trustme:

uv run --with trustme python -m trustme

For production TLS configuration, including mutual TLS and cipher settings, see Configure TLS.

UDP sockets

The User Datagram Protocol (UDP) is a connectionless protocol ideal for high-volume, loss-tolerant data like syslog messages or metrics.

Receive UDP datagrams

Use from_udp to receive UDP messages as structured events:

from_udp "0.0.0.0:514"

Each datagram becomes an event with data (the message content) and peer (the sender’s address) fields.

Parse syslog messages

A common pattern is receiving syslog over UDP:

from_udp "0.0.0.0:514"
this = data.parse_syslog()

Enrich with sender metadata

Include the sender’s IP address and collection timestamp in your events:

from_udp "0.0.0.0:514"
syslog = data.parse_syslog()
this = {
  ...syslog,
  collector: {
    source_ip: peer.ip(),
    received_at: now(),
  },
}

Packet capture

Capture raw network packets from a network interface card (NIC) for deep packet inspection or network forensics.

List available interfaces

Find which network interfaces are available:

nics
select name, addresses, up
where up

{name: "eth0", addresses: ["192.168.1.100", "fe80::1"], up: true}
{name: "lo", addresses: ["127.0.0.1", "::1"], up: true}

Capture packets

Use from_nic to capture and parse packets:

from_nic "eth0" {
  read_pcap
}

Each packet becomes an event with metadata and the raw packet data:

{
  linktype: 1,
  timestamp: 2024-01-15T10:30:45.123456Z,
  captured_packet_length: 74,
  original_packet_length: 74,
  data: "ABY88f1tZJ7zvttmCABFAAA8...",
}

Decapsulate packets

Extract protocol headers from captured packets using the decapsulate function:

from_nic "eth0" {
  read_pcap
}
packet = decapsulate(this)
select packet

{
  packet: {
    ether: {src: "64-9E-F3-BE-DB-66", dst: "00-16-3C-F1-FD-6D", type: 2048},
    ip: {src: "192.168.1.100", dst: "10.0.0.1", type: 6},
    tcp: {src_port: 54321, dst_port: 443},
    community_id: "1:YXWfTYEyYLKVv5Ge4WqijUnKTrM=",
  },
}

The community_id field provides a Community ID hash for correlating network flows across different tools.

Filter by BPF expression

Apply Berkeley Packet Filter (BPF) expressions to capture only specific traffic:

from_nic filter="tcp port 443", "eth0" {
  read_pcap
}

Read PCAP files

The same parsing works for PCAP files:

from_file "capture.pcap" {
  read_pcap
}
packet = decapsulate(this)

Extract flow summaries

Capture packets and extract flow-level summaries:

from_nic "eth0" {
  read_pcap
}
packet = decapsulate(this)
select
  timestamp,
  src_ip=packet.ip.src,
  dst_ip=packet.ip.dst,
  src_port=packet.tcp.src_port? else packet.udp.src_port?,
  dst_port=packet.tcp.dst_port? else packet.udp.dst_port?,
  community_id=packet.community_id