Use lookup tables

This guide shows you how to use lookup tables in Tenzir to store reference data and apply it to OCSF events. Use lookup tables when you have data keyed by an event value, such as a user ID, hostname, IP address, subnet, file hash, or cloud resource ID.

A lookup table is a Tenzir context that stores key-value mappings. Pipelines update the table, and other pipelines query it with context::enrich. This guide covers the basic lookup table lifecycle first, then shows where lookup results should land in OCSF events. Start with the standard enrichments array when the extra data is general context. When the lookup result describes a specific OCSF object, enrich that object instead.

Set up lookup tables

Create lookup tables in separate setup pipelines with context::create_lookup_table, or define them as code by adding them to tenzir.contexts in tenzir.yaml:

<prefix>/etc/tenzir/tenzir.yaml

tenzir:
  contexts:
    user-roles:
      type: lookup-table

This approach is useful when you want the same contexts to exist across deployments.

You can also create lookup tables from the Contexts tab in the platform.

Add entries

Use context::update to add or replace lookup-table entries. This example stores user role metadata keyed by the OCSF user UID:

from {
  user_uid: "S-1-5-21-1001",
  role: "SOC analyst",
  department: "Security Operations",
  privileges: ["investigate", "contain"],
}
context::update "user_roles", key=user_uid, value={
  role: role,
  department: department,
  privileges: privileges,
}

The value can be any structured TQL value. Prefer values that are already close to the OCSF shape you want to produce later. This reduces reshaping in high-volume enrichment pipelines.

Apply lookup tables to OCSF events

OCSF gives you several places to put lookup results. Choose the destination based on what the lookup-table value describes.

Enrich into enrichments

The most general OCSF enrichment target is the base event’s enrichments array. Use it when the lookup result adds useful context but doesn’t belong to a more specific OCSF object.

The following pipeline enriches an OCSF Authentication event with user role context:

from {
  time: 2024-08-22T09:12:34,
  category_uid: 3,
  class_uid: 3002,
  activity_id: 1,
  type_uid: 300201,
  severity_id: 1,
  metadata: {
    version: "1.8.0",
  },
  user: {
    uid: "S-1-5-21-1001",
    name: "alice",
  },
  service: {
    name: "vpn",
  },
  enrichments: [],
}
context::enrich "user_roles",
  key=user.uid,
  into=enrichments,
  mode="append",
  format="ocsf"

{
  time: 2024-08-22T09:12:34,
  category_uid: 3,
  class_uid: 3002,
  activity_id: 1,
  type_uid: 300201,
  severity_id: 1,
  metadata: {
    version: "1.8.0",
  },
  user: {
    uid: "S-1-5-21-1001",
    name: "alice",
  },
  service: {
    name: "vpn",
  },
  enrichments: [{
    created_time: 2024-08-22T09:12:35.069981,
    data: {
      role: "SOC analyst",
      department: "Security Operations",
      privileges: ["investigate", "contain"],
    },
    name: "user.uid",
    provider: "user_roles",
    value: "S-1-5-21-1001",
  }],
}

The format="ocsf" option wraps the lookup result as an OCSF enrichment object with the lookup result in data and the context name in provider. It does not infer an enrichment type. The mode="append" option appends the result to the existing enrichments list instead of overwriting it.

Choose semantic OCSF fields

Use enrichments as the baseline, not as the only destination. If the lookup result describes an OCSF object, write the result into that object’s semantic fields. This keeps the event easier to query and lets downstream tools use the standard schema.

For example, user inventory belongs in user. Create the lookup table in a separate setup pipeline before you populate it:

from {
  uid: "S-1-5-21-1001",
  name: "alice",
  display_name: "Alice Nguyen",
  email_addr: "alice@example.com",
  type_id: 2,
  type: "Admin",
  has_mfa: true,
}
context::update "users", key=uid

Enrich an Authentication event by looking up the source user ID from unmapped and writing the result directly into the OCSF user object:

from {
  time: 2024-08-22T09:12:34,
  category_uid: 3,
  class_uid: 3002,
  activity_id: 1,
  type_uid: 300201,
  severity_id: 1,
  metadata: {
    version: "1.8.0",
  },
  service: {
    name: "vpn",
  },
  unmapped: {
    user_uid: "S-1-5-21-1001",
  },
}
context::enrich "users", key=unmapped.user_uid, into=user

{
  time: 2024-08-22T09:12:34,
  category_uid: 3,
  class_uid: 3002,
  activity_id: 1,
  type_uid: 300201,
  severity_id: 1,
  metadata: {
    version: "1.8.0",
  },
  service: {
    name: "vpn",
  },
  unmapped: {
    user_uid: "S-1-5-21-1001",
  },
  user: {
    uid: "S-1-5-21-1001",
    name: "alice",
    display_name: "Alice Nguyen",
    email_addr: "alice@example.com",
    type_id: 2,
    type: "Admin",
    has_mfa: true,
  },
}

This pattern is the foundation for the OCSF-specific enrichment guides:

• Enrich with asset inventory shows how to enrich src_endpoint, dst_endpoint, device, and related asset fields.
• Enrich with threat intel shows how to enrich enrichments, observables, osint, and related reputation fields.

Use subnet keys

Lookup tables support longest-prefix matching when the stored keys are subnet values and the lookup key is an ip value. This is useful for asset inventory, network ownership, and routing context.

Populate a pre-created subnet table:

from {subnet: 10.0.0.0/22, zone: "corp", owner: {name: "platform"}},
     {subnet: 10.0.0.0/24, zone: "production", owner: {name: "web-platform"}},
     {subnet: 10.0.1.0/24, zone: "database", owner: {name: "data-platform"}}
context::update "network_segments", key=subnet, value={
  subnet_uid: subnet,
  zone: zone,
  owner: owner,
}

Probe the table with endpoint IP addresses from unmapped and write the matched values directly into the OCSF endpoint objects:

from {
  time: 2024-08-22T09:13:01,
  category_uid: 4,
  class_uid: 4001,
  activity_id: 6,
  type_uid: 400106,
  severity_id: 1,
  metadata: {
    version: "1.8.0",
  },
  unmapped: {
    src_ip: 10.0.0.15,
    dst_ip: 10.0.1.20,
  },
}
context::enrich "network_segments", key=unmapped.src_ip, into=src_endpoint
context::enrich "network_segments", key=unmapped.dst_ip, into=dst_endpoint

{
  time: 2024-08-22T09:13:01,
  category_uid: 4,
  class_uid: 4001,
  activity_id: 6,
  type_uid: 400106,
  severity_id: 1,
  metadata: {
    version: "1.8.0",
  },
  unmapped: {
    src_ip: 10.0.0.15,
    dst_ip: 10.0.1.20,
  },
  src_endpoint: {
    subnet_uid: 10.0.0.0/24,
    zone: "production",
    owner: {
      name: "web-platform",
    },
  },
  dst_endpoint: {
    subnet_uid: 10.0.1.0/24,
    zone: "database",
    owner: {
      name: "data-platform",
    },
  },
}

The IP address 10.0.0.15 matches 10.0.0.0/24 instead of 10.0.0.0/22 because /24 is the more specific prefix.

Use compound keys

Use record values as keys when a lookup depends on more than one field. This example populates a pre-created access policy table keyed by a source zone and destination zone:

from {
  key: {
    src_zone: "internet",
    dst_zone: "dmz",
  },
  action: "allow",
  reason: "Public web tier",
},
{
  key: {
    src_zone: "internet",
    dst_zone: "database",
  },
  action: "deny",
  reason: "Database tier is not internet-facing",
}
context::update "access_policies", key=key, value={
  action: action,
  reason: reason,
}

Use the same record shape for enrichment:

from {
  src_endpoint: {
    zone: "internet",
  },
  dst_endpoint: {
    zone: "dmz",
  },
}
context::enrich "access_policies",
  key={
    src_zone: src_endpoint.zone,
    dst_zone: dst_endpoint.zone,
  },
  into=access_policy

{
  src_endpoint: {
    zone: "internet",
  },
  dst_endpoint: {
    zone: "dmz",
  },
  access_policy: {
    action: "allow",
    reason: "Public web tier",
  },
}

Compound keys keep the lookup deterministic when a single event field isn’t selective enough.

Manage table entries

Use context::inspect to view entries:

context::inspect "user_roles"

Use context::erase to remove one entry:

from {user_uid: "S-1-5-21-1001"}
context::erase "user_roles", key=user_uid

Use context::remove to delete the lookup table and its persisted data:

context::remove "user_roles"

Use context::save and context::load to export and import a lookup table.

Export a lookup table to a file:

context::save "user_roles"
to_file "user_roles.bin"

Import a lookup table from a file:

from_file "user_roles.bin"
context::load "user_roles"

Caution

Loading replaces the entire lookup table state. Save the current table first if you need a rollback point.

Expire stale entries

Set timeouts on context::update when the lookup table contains data with a known lifetime, such as active sessions, short-lived indicators, or temporary allowlist entries:

from {
  indicator: "malicious.example",
  confidence: 90,
  source: "ThreatFeed",
}
context::update "active_indicators",
  key=indicator,
  value={
    confidence: confidence,
    source: source,
    updated_time: now(),
  },
  create_timeout=7d,
  write_timeout=24h

create_timeout removes an entry after a fixed lifetime. write_timeout removes an entry if it isn’t refreshed. read_timeout removes an entry if it isn’t used by a lookup.

See Also

• context::create_lookup_table
• context::update
• context::enrich
• context::inspect
• context::erase
• context::save
• context::load
• Enrich with asset inventory
• Enrich with threat intel
• Enrich events with AI
• Enrichment