Type System

Overview

The figure below illustrates the type system at a glance:

type-system

There exist two major classes of types:

  1. Basic: types with static structure with a-priori known representation
  2. Recursive: types that can contain nested types
    • Container: list for value sequences and map for key-value pairs
    • Compound: a record consisting of named fields
    • Alias: named types

Every type has zero or more attributes, which are free-form key-value pairs to further specify type semantics. For example, the #index=hash attribute replaces the default index with a more space-efficient implementation.

Example

To illustrate how the type system interacts with table slices, consider the following example with an event layout (left) along with the corresponding table slice (right):

A table slice layout is always a record type whose fields map to column names. In this example, the record type has the name R and the record has three fields, A, B, and C. The type of A is timestamp, which is defined in VASTs base library and represents the event timestamp. The type of B is string with the #index=hash attribute so this column will use a space efficient hash index. The last event added to this table slice was <2020-01-03, "baz", 4.2>.

Types

In the following we introduce the semantics of the various types, along with identifier and data instantiations that VAST uses in schema definitions and the query language.

Basic Types

IdentifierDescriptionExample Data
noneDenotes an absent or invalid valuenil
boolA boolean valueT, F
intA 64-bit signed integer-42, +3
countA 64-bit unsigned integer0, 42
realA 64-bit double (IEEE 754)-0.7, 1.337
durationA time span (nanosecond granularity)-3us, 24h
timeA time point (nanosecond granularity)now, 1h ago, 2020-01-01+10:42:00
stringA sequence of characters"foo"
patternA regular expression/fo*.bar$/
addrAn IPv4 or IPv6 address::1, 10.0.0.1, 2001:db8::
subnetAn IPv4 or IPv6 subnet::1/128, 10.0.0.0/8, 2001:db8::/32

Container Types

Container types can have zero or more values of a given type, but they count still as single value.

IdentifierDescriptionExample Data
list<T>An (ordered) sequence of values where each element has type T[1, 2, 3], []
map<K, V>An associate array which maps a keys of type K to values of type V{x -> a, y -> b, z -> c}, {}

Compound Types

There exists one compound type: the record type. It is a product type like a C struct or JSON object. By design, every table slice uses a record type to describe its layout. Records consist of one or more fields, each of which have a name and a type.

For example, this is a record with the two fields named flag and host:

record {
flag: bool,
host: addr,
}

Alias Types

In the schema language, alias types make it possible to assign a name to a type. This is commonly used for defining named records, but all types can be aliased including other aliases. Aliases on basic types can be used to allow for more fine grained type extractor lookups. Aliases on records enable reuse of definitions and tie into the --type option of the vast import command.

Type Attributes

A type can have additional key-value pair attributes to enrich its semantics. VAST uses some attributes keys internally, e.g., #index=hash to indicate that the values of a field will be indexed by a hash based index. Attributes have an optional value. For example, both #foo and #foo="bar" is a valid attribute.