Syslog

Tenzir supports parsing and emitting Syslog messages across multiple transport protocols, including both UDP and TCP. This enables seamless integration with Syslog-based systems for ingesting or exporting logs.

Syslog support in Tenzir is powered by two components:

• read_syslog: a parser that turns unstructured Syslog messages into structured events.
• write_syslog: a printer that transforms structured events into compliant Syslog messages.

Together, these building blocks enable round-trip Syslog processing.

Examples

Create a Syslog Server

To receive Syslog messages on a UDP socket, use from_udp:

from_udp "0.0.0.0:514"
this = data.parse_syslog()
publish "syslog"

To use TCP instead of UDP, use load_tcp with read_syslog:

load_tcp "0.0.0.0:514" {
  read_syslog
}
publish "syslog"

The pipeline inside load_tcp executes for each accepted connection.

Parsing CEF, LEEF, or JSON Payloads

If your Syslog messages embed structured formats like CEF, LEEF, or JSON, you can follow up with an additional parser. For example, assume you have a Syslog message that includes CEF:

Nov 13 16:00:02 host123 FOO: CEF:0|FORCEPOINT|Firewall|6.6.1|78002|TLS connection state|0|deviceExternalId=Master FW node 1 dvc=10.1.1.40 dvchost=10.1.1.40 msg=TLS: Couldn't establish TLS connection (11, N/A) deviceFacility=Management rt=Jan 17 2020 08:52:09

When you throw read_syslog at this line, you’ll get this output:

sample.syslog

{
  facility: null,
  severity: null,
  timestamp: "Nov 13 16:00:02",
  hostname: "host123",
  app_name: "FOO",
  process_id: null,
  content: "CEF:0|FORCEPOINT|Firewall|6.6.1|78002|TLS connection state|0|deviceExternalId=Master FW node 1 dvc=10.1.1.40 dvchost=10.1.1.40 msg=TLS: Couldn't establish TLS connection (11, N/A) deviceFacility=Management rt=Jan 17 2020 08:52:09",
}

Note that the content field is just a big string. Parse it with fnparse_cef:

from_file "/tmp/sample.syslog" {
  read_syslog
}
content = content.parse_cef()

This yields the following structured output:

{
  facility: null,
  severity: null,
  timestamp: "Nov 13 16:00:02",
  hostname: "host123",
  app_name: "FOO",
  process_id: null,
  content: {
    cef_version: 0,
    device_vendor: "FORCEPOINT",
    device_product: "Firewall",
    device_version: "6.6.1",
    signature_id: "78002",
    name: "TLS connection state",
    severity: "0",
    extension: {
      deviceExternalId: "Master FW node 1",
      dvc: 10.1.1.40,
      dvchost: 10.1.1.40,
      msg: "TLS: Couldn't establish TLS connection (11, N/A)",
      deviceFacility: "Management",
      rt: "Jan 17 2020 08:52:09",
    },
  },
}

Handling Multi-line Syslog Messages

Tenzir’s Syslog parser supports multi-line messages using a heuristic:

1. Split the input at newlines.
2. Try parsing the next line as a new Syslog message.
3. If successful, treat it as a new message.
4. If parsing fails, append the line to the current message and repeat.

This allows ingesting logs with stack traces or other verbose content correctly.

Preserving the Original Message

For auditing, compliance, or debugging purposes, you may need to keep the original syslog line alongside the parsed fields. Use the raw_message parameter to store the unparsed input:

load_tcp "0.0.0.0:514" {
  read_syslog raw_message=raw
}

This adds a raw field containing the complete original message, including any multi-line content:

{
  facility: 4,
  severity: 2,
  timestamp: "Nov 16 14:55:56",
  hostname: "mymachine",
  app_name: "PROGRAM",
  process_id: null,
  content: "Freeform message",
  raw: "<34>Nov 16 14:55:56 mymachine PROGRAM: Freeform message",
}

RFC 6587 Octet Counting

When receiving syslog over TCP, some implementations use RFC 6587 octet counting to frame messages. Instead of relying on newlines, each message is prefixed with its length in bytes:

65 <165>1 2023-10-11T22:14:15.003Z host app 1234 ID01 - Test message

Here, 65 is the byte count of the syslog message that follows.

Tenzir auto-detects octet-counted messages in both read_syslog for streaming input and fnparse_syslog for parsing individual strings. Use the octet_counting parameter to require or disable this behavior.

RFC 3164 messages with structured data

Some legacy syslog emitters, such as VMware ESXi, prepend RFC 5424-style structured-data blocks to the message content. Tenzir detects these automatically and extracts the key-value pairs into a structured_data field. Events with extracted structured data use the schema syslog.rfc3164.structured while ordinary RFC 3164 events keep the syslog.rfc3164 schema.

Check Point structured data compatibility

Some Check Point exports use structured-data syntax that differs from RFC 5424, for example key:"value" parameters and semicolon separators. Tenzir supports these variants in both read_syslog and parse_syslog.

When a record omits the SD-ID, Tenzir stores the parsed parameters under structured_data.checkpoint_2620.

Emit Events as Syslog

Tenzir also supports creating Syslog messages from structured events via write_syslog.

Here’s a basic example that emits a single Syslog line over UDP:

from {
  facility: 3,
  severity: 6,
  timestamp: 2020-03-02T18:44:46,
  hostname: "parallels-Parallels-Virtual-Platform",
  app_name: "packagekitd",
  process_id: "1370",
  message_id: "",
  structured_data: {},
  message: " PARENT process running...",
}
write_syslog
save_udp "1.2.3.4:514"

This pipeline sends the following RFC 5424-formatted message to 1.2.3.4:514/udp:

<30>1 2020-03-02T18:44:46.000000Z parallels-Parallels-Virtual-Platform packagekitd 1370 - -  PARENT process running...

Example with Structured Data

Here is a richer event with structured Syslog fields. Let’s create a Syslog event from it:

from {
  facility: 20,
  severity: 5,
  version: 8,
  timestamp: 2003-10-11T22:14:15,
  hostname: "mymachineexamplecom",
  app_name: "evntslog",
  process_id: "",
  message_id: "ID47",
  structured_data: {
    "exampleSDID@32473": {
      iut: 5,
      eventSource: "Applic\\ation",
      eventID: 1011,
    },
    "examplePriority@32473": {
      class: "high",
    },
  },
  message: null,
}
write_syslog

Output:

<165>1 2003-10-11T22:14:15.000000Z mymachineexamplecom evntslog - ID47 [exampleSDID@32473 iut="5" eventSource="Applic\\ation" eventID="1011"][examplePriority@32473 class="high"]

The write_syslog operator converts the structured_data field into a valid RFC 5424 structured block.