Transform collections

Lists and records are fundamental data structures in TQL. This guide shows you how to work with these collections - accessing elements, transforming values, and combining data structures.

Work with lists

Lists (arrays) contain ordered sequences of values. Let’s explore how to manipulate them.

Access list elements

Get values from specific positions:

from {items: ["first", "second", "third", "fourth"]}
set first_item = items[0]
set last_item = items[-1]
set length = items.length()

{
  items: ["first", "second", "third", "fourth"],
  first_item: "first",
  last_item: "fourth",
  length: 4
}

Index notation:

• [0] - First element (0-based indexing)
• [-1] - Last element (negative indices count from end)
• .length() - Get the number of elements

Add elements to lists

Use append() and prepend():

from {colors: ["red", "green"]}
set with_blue = colors.append("blue")
set with_yellow = with_blue.prepend("yellow")
set multi_append = colors.append("blue").append("purple")

{
  colors: ["red", "green"],
  with_blue: ["red", "green", "blue"],
  with_yellow: ["yellow", "red", "green", "blue"],
  multi_append: ["red", "green", "blue", "purple"]
}

Combine lists

Join multiple lists with concatenate() or spread syntax:

from {
  list1: [1, 2, 3],
  list2: [4, 5, 6],
  list3: [7, 8, 9]
}
set combined = concatenate(concatenate(list1, list2), list3)
set spread = [...list1, ...list2, ...list3]
set with_value = [...list1, 10, ...list2]

{
  list1: [1, 2, 3],
  list2: [4, 5, 6],
  list3: [7, 8, 9],
  combined: [1, 2, 3, 4, 5, 6, 7, 8, 9],
  spread: [1, 2, 3, 4, 5, 6, 7, 8, 9],
  with_value: [1, 2, 3, 10, 4, 5, 6]
}

Transform list elements

Apply functions to each element with map():

from {
  prices: [10, 20, 30],
  names: ["alice", "bob", "charlie"]
}
set with_tax = prices.map(p => p * 1.1)
set uppercase = names.map(n => n.to_upper())
set squared = prices.map(x => x * x)

{
  prices: [10, 20, 30],
  names: ["alice", "bob", "charlie"],
  with_tax: [11.0, 22.0, 33.0],
  uppercase: ["ALICE", "BOB", "CHARLIE"],
  squared: [100, 400, 900]
}

Filter list elements

Keep only elements that match a condition with where():

from {
  numbers: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
  users: ["alice", "bob", "anna", "alex"]
}
// Note: The modulo operator (%) is not currently supported in TQL
// Here's an alternative approach for filtering:
set big_nums = numbers.where(n => n > 5)
set small_nums = numbers.where(n => n <= 5)
set a_names = users.where(u => u.starts_with("a"))

{
  numbers: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
  users: ["alice", "bob", "anna", "alex"],
  big_nums: [6, 7, 8, 9, 10],
  small_nums: [1, 2, 3, 4, 5],
  a_names: ["alice", "anna", "alex"]
}

Sort lists

Order elements with sort():

from {
  numbers: [3, 1, 4, 1, 5, 9],
  words: ["zebra", "apple", "banana"]
}
set sorted_nums = numbers.sort()
set sorted_words = words.sort()
// Note: reverse() is only for strings, not lists
// To reverse a list, you would need to use the reverse operator in a pipeline

{
  numbers: [3, 1, 4, 1, 5, 9],
  words: ["zebra", "apple", "banana"],
  sorted_nums: [1, 1, 3, 4, 5, 9],
  sorted_words: ["apple", "banana", "zebra"]
}

Get unique values

Remove duplicates with distinct():

from {
  items: ["a", "b", "a", "c", "b", "d"],
  numbers: [1, 2, 2, 3, 3, 3, 4]
}
set unique_items = distinct(items)
set unique_nums = distinct(numbers)

{
  items: ["a", "b", "a", "c", "b", "d"],
  numbers: [1, 2, 2, 3, 3, 3, 4],
  unique_items: ["a", "b", "c", "d"],
  unique_nums: [1, 2, 3, 4]
}

Flatten nested lists

Note: Direct list flattening is not currently supported in TQL. The flatten() function is designed for flattening records, not lists. To work with nested lists, you would need to process them element by element.

Work with records

Records (objects) contain key-value pairs. Here’s how to manipulate them.

Access record fields

Get values using dot notation or brackets:

from {
  user: {
    name: "Alice",
    age: 30,
    address: {
      city: "NYC",
      zip: "10001"
    }
  }
}
set name = user.name
set city = user.address.city
set zip = user["address"]["zip"]
set has_email = user.has("email")

{
  user: {
    name: "Alice",
    age: 30,
    address: {city: "NYC", zip: "10001"}
  },
  name: "Alice",
  city: "NYC",
  zip: "10001",
  has_email: false
}

Get keys and values

Extract field names and values:

from {
  config: {
    host: "localhost",
    port: 8080,
    ssl: true
  }
}
set field_names = config.keys()
// Note: values() function is not available
set num_fields = config.keys().length()

{
  config: {host: "localhost", port: 8080, ssl: true},
  field_names: ["host", "port", "ssl"],
  num_fields: 3
}

Merge records

Combine multiple records with merge() or spread syntax:

from {
  defaults: {host: "localhost", port: 80, ssl: false},
  custom: {port: 8080, ssl: true}
}
set merged = merge(defaults, custom)
set spread = {...defaults, ...custom}
set with_extra = {...defaults, ...custom, debug: true}

{
  defaults: {host: "localhost", port: 80, ssl: false},
  custom: {port: 8080, ssl: true},
  merged: {host: "localhost", port: 8080, ssl: true},
  spread: {host: "localhost", port: 8080, ssl: true},
  with_extra: {host: "localhost", port: 8080, ssl: true, debug: true}
}

Transform record values

Note: The map_values function is not available in TQL. To transform record values, you would need to reconstruct the record with transformed values manually:

from {
  prices: {
    apple: 1.50,
    banana: 0.75,
    orange: 2.00
  }
}
// Manual transformation example:
set with_tax = {
  apple: prices.apple * 1.1,
  banana: prices.banana * 1.1,
  orange: prices.orange * 1.1
}

Filter record fields

Keep only specific fields:

from {
  user: {
    id: 123,
    name: "Alice",
    email: "alice@example.com",
    password: "secret",
    api_key: "xyz123"
  }
}
// Note: filter_keys and select functions are not available
// Manual field selection:
set public_info = {
  id: user.id,
  name: user.name,
  email: user.email
}
set contact = {
  name: user.name,
  email: user.email
}

{
  user: {
    id: 123,
    name: "Alice",
    email: "alice@example.com",
    password: "secret",
    api_key: "xyz123"
  },
  public_info: {
    id: 123,
    name: "Alice",
    email: "alice@example.com"
  },
  contact: {
    name: "Alice",
    email: "alice@example.com"
  }
}

Combine lists and records

Work with collections of records:

from {
  users: [
    {name: "Alice", age: 30, city: "NYC"},
    {name: "Bob", age: 25, city: "SF"},
    {name: "Charlie", age: 35, city: "NYC"}
  ]
}
set names = users.map(u => u.name)
set nyc_users = users.where(u => u.city == "NYC")
set avg_age = users.map(u => u.age).sum() / users.length()

{
  users: [
    {name: "Alice", age: 30, city: "NYC"},
    {name: "Bob", age: 25, city: "SF"},
    {name: "Charlie", age: 35, city: "NYC"}
  ],
  names: ["Alice", "Bob", "Charlie"],
  nyc_users: [
    {name: "Alice", age: 30, city: "NYC"},
    {name: "Charlie", age: 35, city: "NYC"}
  ],
  avg_age: 30.0
}

Advanced transformations

Zip lists together

Combine parallel lists with zip():

from {
  names: ["Alice", "Bob", "Charlie"],
  ages: [30, 25, 35],
  cities: ["NYC", "SF", "LA"]
}
// Note: zip only takes 2 arguments, returns records with left/right fields
set name_age = zip(names, ages)
set zipped = zip(name_age, cities)
set users = zipped.map(z => {
  name: z.left.left,
  age: z.left.right,
  city: z.right
})

{
  names: ["Alice", "Bob", "Charlie"],
  ages: [30, 25, 35],
  cities: ["NYC", "SF", "LA"],
  name_age: [
    {left: "Alice", right: 30},
    {left: "Bob", right: 25},
    {left: "Charlie", right: 35}
  ],
  zipped: [
    {left: {left: "Alice", right: 30}, right: "NYC"},
    {left: {left: "Bob", right: 25}, right: "SF"},
    {left: {left: "Charlie", right: 35}, right: "LA"}
  ],
  users: [
    {name: "Alice", age: 30, city: "NYC"},
    {name: "Bob", age: 25, city: "SF"},
    {name: "Charlie", age: 35, city: "LA"}
  ]
}

Enumerate with indices

Add row numbers to your data using the enumerate operator:

from {item: "apple"}, {item: "banana"}, {item: "cherry"}
enumerate row

{row: 0, item: "apple"}
{row: 1, item: "banana"}
{row: 2, item: "cherry"}

This is useful for tracking position in sequences or creating unique identifiers for each event.

Practical examples

Extract and transform nested data

from {
  response: {
    status: 200,
    data: {
      users: [
        {id: 1, name: "Alice", scores: [85, 92, 88]},
        {id: 2, name: "Bob", scores: [78, 81, 85]}
      ]
    }
  }
}
set users = response.data.users
set summaries = users.map(u, {
  name: u.name,
  avg_score: u.scores.sum() / u.scores.length(),
  max_score: u.scores.max()
})

{
  response: {...},
  users: [
    {id: 1, name: "Alice", scores: [85, 92, 88]},
    {id: 2, name: "Bob", scores: [78, 81, 85]}
  ],
  summaries: [
    {name: "Alice", avg_score: 88.33333333333333, max_score: 92},
    {name: "Bob", avg_score: 81.33333333333333, max_score: 85}
  ]
}

Work with indexed data

Create lookups by extracting specific fields:

from {
  items: [
    {id: "A001", name: "Widget", price: 10},
    {id: "B002", name: "Gadget", price: 20},
    {id: "C003", name: "Tool", price: 15}
  ]
}
set first_item = items.first()
set ids = items.map(item => item.id)
set names = items.map(item => item.name)
set expensive = items.where(item => item.price > 15)

{
  items: [...],
  first_item: {id: "A001", name: "Widget", price: 10},
  ids: ["A001", "B002", "C003"],
  names: ["Widget", "Gadget", "Tool"],
  expensive: [
    {id: "B002", name: "Gadget", price: 20}
  ]
}

Best practices

1. Choose the right structure: Use lists for ordered data, records for named fields
2. Avoid deep nesting: Flatten structures when possible for easier access
3. Use functional methods: Prefer map(), filter(), etc. over manual loops
4. Handle empty collections: Check length before accessing elements
5. Preserve immutability: Collection functions return new values, not modify existing

Related guides

• Filter and select data - Filter entire event streams
• Transform basic values - Work with simple types
• Shape data - Overview of all shaping operations