Skip to main content
Version: v4.18

flatten

Flattens nested data.

Synopsis

flatten [<separator>]

Description

The flatten operator acts on container types:

  1. Records: Join nested records with a separator (. by default). For example, if a field named x is a record with fields a and b, flattening will lift the nested record into the parent scope by creating two new fields x.a and x.b.
  2. Lists: Merge nested lists into a single (flat) list. For example, [[[2]], [[3, 1]], [[4]]] becomes [2, 3, 1, 4].

For records inside lists, flatten "pushes lists down" into one list per record field. For example, the record

{
"foo": [
{
"a": 2,
"b": 1
},
{
"a": 4
}
]
}

becomes

{
"foo.a": [2, 4],
"foo.b": [1, null]
}

Lists nested in records that are nested in lists will also be flattened. For example, the record

{
"foo": [
{
"a": [
[2, 23],
[1,16]
],
"b": [1]
},
{
"a": [[4]]
}
]
}

becomes

{
"foo.a": [
2,
23,
1,
16,
4
],
"foo.b": [
1
]
}

As you can see from the above examples, flattening also removes null values.

<separator>

The separator string to join the field names of nested records.

Defaults to ..

Examples

Consider the following record:

{
"src_ip": "147.32.84.165",
"src_port": 1141,
"dest_ip": "147.32.80.9",
"dest_port": 53,
"event_type": "dns",
"dns": {
"type": "query",
"id": 553,
"rrname": "irc.freenode.net",
"rrtype": "A",
"tx_id": 0,
"grouped": {
"A": ["tenzir.com", null]
}
}
}

After flatten the record looks as follows:

{
"src_ip": "147.32.84.165",
"src_port": 1141,
"dest_ip": "147.32.80.9",
"dest_port": 53,
"event_type": "dns",
"dns.type": "query",
"dns.id": 553,
"dns.rrname": "irc.freenode.net",
"dns.rrtype": "A",
"dns.tx_id": 0,
"dns.grouped.A": ["tenzir.com"]
}

Note that dns.grouped.A no longer contains a null value.