OCSF 1.6.0

This release adds support for OCSF 1.6.0 and introduces the replace operator.

Download the release on GitHub.

Features

Check if your data is truly empty

Ever stared at your security logs wondering if that suspicious-looking field is actually empty or just pretending? We’ve all been there. That’s why we added is_empty() - a universal emptiness detector that works on strings, lists, and records.

from {
  id: "evt-12345",
  tags: ["malware", "c2"],
  metadata: {},
  description: "",
  iocs: []
}
has_metadata = not metadata.is_empty()
has_description = not description.is_empty()
has_iocs = not iocs.is_empty()

{
  id: "evt-12345",
  tags: [
    "malware",
    "c2",
  ],
  metadata: {},
  description: "",
  iocs: [],
  has_metadata: false,
  has_description: false,
  has_iocs: false,
}

No more checking length() == 0 or wondering if that field exists but is empty. Just ask is_empty() and move on with your threat hunting!

By @mavam in #5403.

Replacing values

The new replace operator allows you to find and replace all occurrences of a specific value across all fields (but not in lists) in your data with another value. This is particularly useful for data sanitization, redacting sensitive information, or normalizing values across datasets.

The operator scans every field in each input event and replaces any value that equals the what parameter with the value specified by with.

Replace all occurrences of the string "-" with null:

from {
  status: "-",
  data: {result: "-", count: 42},
  items: ["-", "valid", "-"]
}
replace what="-", with=null

{
  status: null,
  data: {result: null, count: 42},
  items: ["-", "valid", "-"]
}

Redact a specific IP address across all fields:

from {
  src_ip: 192.168.1.1,
  dst_ip: 10.0.0.1,
  metadata: {source: 192.168.1.1}
}
replace what=192.168.1.1, with="REDACTED"

{
  src_ip: "REDACTED",
  dst_ip: 10.0.0.1,
  metadata: {source: "REDACTED"}
}

By @raxyte in #5372.

role and external_id options for `load_s3

We added role and external_id options to the load_s3 operator, bringing it in line with save_s3, which already features these options.

By @IyeOnline in #5406.

Support for OCSF 1.6.0

The ocsf:: family of operators now supports OCSF 1.6.0.

Furthermore, the version 1.7.0-dev is also supported now.

By @jachris in #5407.

Changes

Handling of type conflicts when reading data

Tenzir requires all items of a list to have the same type. As a result, items in lists that contain different types (such as [1, "test"]) are cast to the common type string. Previously, all items were stored with their JSON representation, leading to the result ["1", "\"test\""]. Now, only lists and record are stored as JSON, and strings are preserved without extra quotes. Thus, the new output is ["1", "test"].

By @jachris in #5405.

Automatic integer casting in ocsf::apply

The ocsf::apply operator now automatically casts uint64 values to int64 when the OCSF schema expects an integer field. This is important because the exact integer type is mostly considered an implementation detail. Unsigned integers are mainly produced when reading events for which a schema has been defined. This change makes sure that OCSF mappings that use the resulting events can successfully pass through ocsf::apply.

Example:

from {
  class_uid: 4001,
  metadata: {
    version: "1.5.0"
  },
  severity_id: uint(3)
}
ocsf::apply

Previously, this would result in a type mismatch warning and the severity_id field would be set to null. Now the uint64 value 3 is automatically cast to int64, preserving the data. Values that exceed the maximum int64 value will still generate a warning and be set to null.

By @jachris in #5401.

Bug Fixes

Metadata handling of ocsf::derive and ocsf::trim

The ocsf::derive and ocsf::trim operators now correctly preserve the metadata (such as @name) of the incoming event instead of overwriting it with the internal metadata used to encode OCSF schemas.

By @jachris in #5402.

Optimization of the delay operator

The delay operator optimization routine incorrectly declared that the behavior of the operator does not depend on the order of its input. As a result, chains such as sort -> delay -> publish did not correctly delay events.

By @jachris in #5399.