Skip to main content
Version: v4.21

Type System

One design goal of Tenzir is being expressive enough to capture the semantics of the domain. This led us to develop a rich type system for structured security data, inspired by the Zeek network security monitor.

Terminology

The diagram below illustrates the type system at a glance:

Basic Types

Basic types are stateless types with a static structure. Their representation is known a-priori.

The following tables summarizes the basic types.

TypeDescription
noneDenotes an absent or invalid value
boolA boolean value
int64A 64-bit signed integer
uint64A 64-bit unsigned integer
doubleA 64-bit double (IEEE 754)
durationA time span (nanosecond granularity)
timeA time point (nanosecond granularity)
stringA UTF-8 encoded string
blobAn arbitrary sequence of bytes
patternA regular expression
ipAn IPv4 or IPv6 address
subnetAn IPv4 or IPv6 subnet
Experimental

The blob type is still experimental and not yet fully supported.

Complex Types

Complex types are stateful types that carry additional runtime information.

Enumeration

The enum type is a list of predefined string values. It comes in handy for low-cardinality values from a fixed set of options.

Tenzir implements an enum as an Arrow Dictionary.

List

The list type is an ordered sequence of values with a fixed element type.

Lists have zero or more elements.

Record

The record type consists of an ordered sequence fields, each of which have a name and type. Records must have at least one field.

The field name is an arbitrary UTF-8 string.

The field type is any Tenzir type.

Optionality

All types are optional in that there exists an additional null data point in every value domain. Consequently, Tenzir does not have a special type to indicate optionality.

Attributes

Every type has zero or more attributes, which are free-form key-value pairs to enrich types with custom semantics.

For duration types, the attribute unit can be set to customize how numeric values are interpreted by parsers. For example, #unit=s makes it so that the number 1708430755 is parsed as "1708430755s". The unit attribute can also be set for timestamps, which means that numeric values are interpreted as offsets from the Unix epoch. For example, #unit=s makes it so that 1708430755 is parsed as "2023-02-20T12:05:55".

Aliases

An alias wraps an existing type under a new name. Aliases are first-class types, meaning you can also attach separate attributes to them.

All alias types have a name. They coexist in a global namespace.

Alias = Specialization

An alias always refines the type it points to, i.e., it is more specific that its parent. For example, let's assume a type U is an alias for an existing type T. Then, the type extractor :U only resolves for types of instance U. However, :T comprises both instances of types U and T.

Comparison to Arrow

All Tenzir types have a lossless mapping to Arrow types, however, not all Arrow types have a Tenzir equivalent. As a result, it is currently not yet possible to import arbitrary Arrow data. In the future, we plan to extend our support for Arrow-native types and also offer conversion options for seamless data handover.

Tenzir has a few domain-specific types that map to Arrow extension types. These are currently:

  • enum
  • ip
  • subnet

In Tenzir, the type is the entity that contains metadata. In Arrow, it's the record field or the schema.

More on Arrow & Tenzir

If you want to learn more about why Tenzir uses Apache Arrow, please read our two blog posts that explain why we build on top of Arrow.