Shape data

TQL comes with numerous transformation operators that change the shape of events, as well as functions that work on values within a single event.

This guide provides an overview of data shaping capabilities in TQL, showcasing the operators and functions that you need for day-to-day data transformation tasks.

Understanding operators vs functions

Before diving into specific operations, it’s important to understand a key distinction in TQL:

Operators work on streams of events and can keep state between multiple events (e.g., where, select, summarize)
Functions work on individual values within a single event (e.g., starts_with(), round(), merge())

Here is a visual overview of transformations that you can perform over a stream of events:

Common data shaping tasks

Here are the most common data shaping operations organized by use case:

Filter and select data

The fundamental operations for controlling which events and fields flow through your pipeline. See the Filter and select data guide for detailed examples of:

Filtering events with where
Selecting fields with select
Removing fields with drop
Adding fields with set

Control event flow

Manage how many events pass through your pipeline and in what order. See the Slice and sample data guide for:

Getting first/last events with head and tail
Slicing ranges with slice
Sampling by schema with taste
Reversing order with reverse

Transform values

Convert and manipulate individual values within events. See the Transform basic values guide for:

Type conversions (int(), float(), string(), time())
String operations (to_upper(), trim(), capitalize())
Mathematical operations (round(), abs(), sqrt())
Handling null values with the else keyword

Work with complex structures

Relocate fields with `move`

Use the move operator to rename and relocate fields in one operation:

from {old: 42}
move new = old

{new: 42}

Moving multiple fields:

from {foo: 1, bar: 2}
move foo=bar, qux=foo

{
  foo: 2,
  qux: 1,
}

Aggregate events with `summarize`

Use summarize to group and aggregate data. This example groups events by the y field and sums up the x values for each group:

from {x: 0, y: 0, z: 1},
     {x: 1, y: 1, z: 2},
     {x: 1, y: 1, z: 3}
summarize y, x=sum(x)

{y: 0, x: 0}
{y: 1, x: 2}

In this example:

Events are grouped by the value of y (0 or 1)
For y=0, there’s one event with x=0, so sum(x)=0
For y=1, there are two events with x=1 each, so sum(x)=2

See the aggregation functions reference for all available aggregation functions like count(), mean(), max(), etc.

Reorder events with `sort`

Use sort to arrange events by field values:

from {x: 2, y: "bar"},
     {x: 3, y: "baz"},
     {x: 1, y: "foo"}
sort -x

{x: 3, y: "baz"}
{x: 2, y: "bar"}
{x: 1, y: "foo"}

Prepending the field with - reverses the sort order.

Break up lists with `unroll`

Use unroll to expand lists into separate events:

from {
  xs: [{a: 1}, {a: 2}],
  y: "foo",
}
unroll xs

{
  xs: {
    a: 1,
  },
  y: "foo",
}
{
  xs: {
    a: 2,
  },
  y: "foo",
}

Manipulate records and lists

Combine records with `merge`

Use the merge function to combine records. This is useful when you need to consolidate data from multiple sources:

from {
  foo: {
    bar: 1,
    baz: 2,
  },
  qux: {
    fred: 3,
    george: 4,
    bar: 5,
  }
}
set this = merge(foo, qux)

{
  bar: 5,
  baz: 2,
  fred: 3,
  george: 4
}

Note that the field bar appears in both records. The value from the second argument (qux.bar = 5) overwrites the value from the first (foo.bar = 1).

You can also use the spread expression as shorthand:

set this = {...foo, ...qux}

Combine lists with `concatenate`

Use concatenate to join lists:

from {
  xs: [1,2,3],
  ys: [4,5,6],
}
select result = concatenate(xs, ys)

{
  result: [
    1,
    2,
    3,
    4,
    5,
    6,
  ],
}

Or use the spread expression:

select result = [...xs, ...ys]

Add values to lists

Use append and prepend:

from {
  xs: [2],
}
set xs = append(xs, 3)
set xs = prepend(xs, 1)

{
  xs: [
    1,
    2,
    3,
  ],
}

Specialized operations

Perform bitwise operations

TQL provides bitwise functions for low-level data manipulation using bit_and(), bit_or(), bit_xor(), bit_not(), shift_left(), and shift_right():

from {
  band: bit_and(5, 3),
  bor: bit_or(5, 3),
  bxor: bit_xor(5, 3),
  bnot: bit_not(5),
  shl: shift_left(5, 2),
  shr: shift_right(5, 1),
}

{
  band: 1,  // (0101 & 0011 = 0001)
  bor: 7,   // (0101 | 0011 = 0111)
  bxor: 6,  // (0101 ^ 0011 = 0110)
  bnot: -6, // (~0101 = 1010)
  shl: 20,  // (0101 << 2 = 10100)
  shr: 2,   // (0101 >> 1 = 0010)
}

Best practices

Understand the operator/function distinction: Use operators for stream-level operations and functions for value-level transformations.
Filter early: Apply where conditions as early as possible to reduce data volume.
Select only what you need: Use select to keep only necessary fields, especially with large events.
Choose the right tool:
- Use select to keep specific fields
- Use drop to remove a few fields from many
- Use set to add fields without changing existing ones
- Use move to rename fields efficiently
Be mindful of performance: Some operators like tail and reverse must buffer all input before producing output.

Where to go from here

Explore these specialized guides for deeper coverage of specific topics:

Filter and select data - Master filtering and field selection
Transform basic values - Type conversions and value manipulation
Manipulate strings - Text processing and formatting
Work with time - Parse, format, and calculate with timestamps
Transform collections - Work with lists and records
Aggregate and summarize - Statistical operations and grouping
Slice and sample data - Control event flow
Extract structured data from text - Parse complex text formats
Convert data formats - Transform between JSON, CSV, YAML, and more
Reshape complex data - Advanced structural transformations
Deduplicate events - Remove duplicate events

For a complete list of available operators and functions, see the operators and functions reference documentation.