Skip to content

Generate a parser

This guide shows you how to generate a TQL parser from sample log events using the MCP server’s code generation tools.

What you’ll create

Section titled “What you’ll create”

A complete parser package that:

  • Detects the log format automatically (e.g., JSON, CSV, syslog, key-value)
  • Generates TQL parsing code with proper operators and functions
  • Infers types and creates schema transformations
  • Includes test cases with sample data
  • Provides a ready-to-install Tenzir package

Prerequisites

Section titled “Prerequisites”

Generate the parser

Section titled “Generate the parser”

  1. Provide sample log events. Give your AI agent one or more representative log events:

    Prompt with Fortinet DNS query log
    Hey, generate a parser for this log sample:
    

    date=2019-05-15 time=15:05:49 logid="1501054802" type="utm" subtype="dns" eventtype="dns-response" level="notice" vd="vdom1" eventtime=1557957949740931155 policyid=1 sessionid=6887 srcip=10.1.100.22 srcport=50002 srcintf="port12" srcintfrole="undefined" dstip=172.16.100.100 dstport=53 dstintf="port11" dstintfrole="undefined" proto=17 profile="dnsfilter_fgd" srcmac="a2:e9:00:ec:40:41" xid=57945 qname="changelogs.ubuntu.com" qtype="AAAA" qtypeval=28 qclass="IN" ipaddr="2001:67c:1560:8008::11" msg="Domain is monitored" action="pass" cat=52 catdesc="Information Technology"

    The MCP server uses the make_parser tool to analyze the format and generate a complete workflow with step-by-step instructions.

  2. Wait for completion. In most cases you can now keep the hands off the keyboard and wait until completion. The make_parser tool provides your agent with detailed guidance for:

    • Format detection: Identifying the log format and structure
    • Operator selection: Choosing the right TQL operators for parsing
    • Type inference: Creating proper schema transformations
    • Parser generation: Generating the complete TQL parsing code
    • Testing: Creating test cases to validate the parser Your AI agent follows these instructions to build the parser.

  3. Verify and iterate. The above log sample resulted in a new package fortinet in your agent’s current working directory. Review its contents and validate both the parser and the tests before you ship the package.

The generated package may look like this:

  • Directoryfortinet/
    • Directoryoperators/
      • parse.tql
    • Directorytests/
      • Directoryinputs/
        • combined.log
        • sample-synthetic.log
        • sample.log
      • parse.tql
      • parse.txt
    • package.yaml

This is a valid package including tests. Make sure that you check operators/parse.tql that defines fortinet::parse and inspect tests/parse.txt for the expected output.

Next steps

Section titled “Next steps”
  • Refine the parser: Adjust field types, add transformations, or handle edge cases.
  • Install the package: Use package::add or set tenzir.package_dirs to make the parser package available in your Tenzir node.
  • Map to OCSF: Proceed with adding an OCSF mapping to normalize the parsed data.
  • Add a changelog entry: Ask the agent to add a changelog entry, documenting the work.

Last updated: