Transform basic values

Transforming values is a fundamental part of data processing. This guide shows you how to convert between different data types, perform basic calculations, and manipulate simple values within your events.

Type conversions

TQL provides functions to convert values between different types. This is essential when data arrives in the wrong format or when you need specific types for further processing.

Convert to numbers

Use int() and float() to convert values to numeric types:

from {price: "42", quantity: "3.5"},
     {price: "99", quantity: "1.0"}
price = price.int()
quantity = quantity.float()

{price: 42, quantity: 3.5}
{price: 99, quantity: 1.0}

Convert to strings

Use string() to convert any value to its string representation:

from {status: 200, ratio: 0.95},
     {status: 404, ratio: 0.05}
message = status.string() + " - " + (ratio * 100).string() + "%"

{status: 200, ratio: 0.95, message: "200 - 95.0%"}
{status: 404, ratio: 0.05, message: "404 - 5.0%"}

Parse times and durations

Convert strings to time values with time():

from {timestamp: "2024-01-15"},
     {timestamp: "2024-02-20"}
parsed_time = timestamp.time()

{timestamp: "2024-01-15", parsed_time: 2024-01-15T00:00:00Z}
{timestamp: "2024-02-20", parsed_time: 2024-02-20T00:00:00Z}

Convert strings to durations with duration():

from {interval: "5s"},
     {interval: "2.5min"}
parsed_duration = interval.duration()

{interval: "5s", parsed_duration: 5s}
{interval: "2.5min", parsed_duration: 2.5min}

Convert to unsigned integers

Use uint() for non-negative integers:

from {count: "42", ratio: 3.7},
     {count: "-5", ratio: 2.3}
count_uint = count.uint()
ratio_uint = ratio.uint()

{count: "42", ratio: 3.7, count_uint: 42, ratio_uint: 3}
{count: "-5", ratio: 2.3, count_uint: null, ratio_uint: 2}

This pipeline elicits the following warning:

warning: `uint` failed to convert some string
 --> /tmp/pipeline.tql:3:14
  |
3 | count_uint = count.uint()
  |              ~~~~~
  |

Work with IP addresses and subnets

TQL supports IP address and subnet literals directly. You can also parse them from strings using ip() and subnet():

from {direct_ip: 192.168.1.1, direct_subnet: 10.0.0.0/24},
     {direct_ip: ::1, direct_subnet: 2001:db8::/32}
ipv6_check = direct_ip.is_v6()

{
  direct_ip: 192.168.1.1,
  direct_subnet: 10.0.0.0/24,
  ipv6_check: false,
}
{
  direct_ip: ::1,
  direct_subnet: 2001:db8::/32,
  ipv6_check: true,
}

Parse from strings when needed:

from {client: "192.168.1.1", network: "10.0.0.0/24"},
     {client: "10.0.0.5", network: "192.168.0.0/16"}
client_ip = client.ip()
network_subnet = network.subnet()

{
  client: "192.168.1.1",
  network: "10.0.0.0/24",
  client_ip: 192.168.1.1,
  network_subnet: 10.0.0.0/24,
}
{
  client: "10.0.0.5",
  network: "192.168.0.0/16",
  client_ip: 10.0.0.5,
  network_subnet: 192.168.0.0/16,
}

IP address inspection

Analyze and categorize IP addresses with inspection functions:

Check IP address types

Use IP inspection functions like is_v4(), is_v6(), is_private(), is_global(), is_loopback(), and is_multicast() to analyze addresses:

from {ip1: 192.168.1.1, ip2: 8.8.8.8, ip3: ::1},
     {ip1: 10.0.0.1, ip2: 224.0.0.1, ip3: 2001:db8::1}
is_v4 = ip1.is_v4()
is_v6 = ip3.is_v6()
is_private = ip1.is_private()
is_global = ip2.is_global()
is_loopback = ip3.is_loopback()
is_multicast = ip2.is_multicast()

{
  ip1: 192.168.1.1,
  ip2: 8.8.8.8,
  ip3: ::1,
  is_v4: true,
  is_v6: true,
  is_private: true,
  is_global: true,
  is_loopback: true,
  is_multicast: false,
}
{
  ip1: 10.0.0.1,
  ip2: 224.0.0.1,
  ip3: 2001:db8::1,
  is_v4: true,
  is_v6: true,
  is_private: true,
  is_global: false,
  is_loopback: false,
  is_multicast: true,
}

Categorize IP addresses

Get detailed IP address classification with ip_category():

from {client: "192.168.1.100", server: "8.8.8.8", local: "127.0.0.1"},
     {client: "10.0.0.5", server: "224.0.0.251", local: "::1"}
client_category = client.ip().ip_category()
server_category = server.ip().ip_category()
local_category = local.ip().ip_category()

{
  client: "192.168.1.100",
  server: "8.8.8.8",
  local: "127.0.0.1",
  client_category: "private",
  server_category: "global",
  local_category: "loopback",
}
{
  client: "10.0.0.5",
  server: "224.0.0.251",
  local: "::1",
  client_category: "private",
  server_category: "multicast",
  local_category: "loopback",
}

Check link-local addresses

Identify link-local addresses with is_link_local():

from {addr1: 169.254.1.1, addr2: fe80::1, addr3: 192.168.1.1},
     {addr1: 169.254.0.1, addr2: 2001:db8::1, addr3: 10.0.0.1}
link_local1 = addr1.is_link_local()
link_local2 = addr2.is_link_local()
link_local3 = addr3.is_link_local()

{
  addr1: 169.254.1.1,
  addr2: fe80::1,
  addr3: 192.168.1.1,
  link_local1: true,
  link_local2: true,
  link_local3: false,
}
{
  addr1: 169.254.0.1,
  addr2: 2001:db8::1,
  addr3: 10.0.0.1,
  link_local1: true,
  link_local2: false,
  link_local3: false,
}

Basic string operations

Transform strings with simple operations to clean and standardize your data.

Change case

Convert strings to different cases:

from {name: "alice smith", code: "xyz"},
     {name: "BOB JONES", code: "ABC"}
name = name.to_title()
code = code.to_upper()

{name: "Alice Smith", code: "XYZ"}
{name: "Bob Jones", code: "ABC"}

Trim whitespace

Remove unwanted whitespace from strings:

from {input: "  hello  ", data: "world   "},
     {input: "   test", data: "  value  "}
input = input.trim()
data = data.trim()

{input: "hello", data: "world"}
{input: "test", data: "value"}

Capitalize strings

Capitalize the first letter of a string:

from {word: "hello", phrase: "good morning"},
     {word: "world", phrase: "how are you"}
word = word.capitalize()

{word: "Hello", phrase: "good morning"}
{word: "World", phrase: "how are you"}

Mathematical operations

Perform calculations on numeric values within your events.

Basic arithmetic

Use standard arithmetic operators:

from {a: 10, b: 3},
     {a: 20, b: 4}
sum = a + b
diff = a - b
product = a * b
quotient = (a / b).int()

{a: 10, b: 3, sum: 13, diff: 7, product: 30, quotient: 3}
{a: 20, b: 4, sum: 24, diff: 16, product: 80, quotient: 5}

Rounding numbers

Round numbers to specific precision:

from {value: 3.14159},
     {value: 2.71828}
rounded = value.round()
ceil_val = value.ceil()
floor_val = value.floor()

{value: 3.14159, rounded: 3, ceil_val: 4, floor_val: 3}
{value: 2.71828, rounded: 3, ceil_val: 3, floor_val: 2}

Mathematical functions

Use abs() for absolute values and sqrt() for square roots:

from {x: -5, y: 16},
     {x: -10, y: 25}
abs_x = x.abs()
sqrt_y = y.sqrt()

{
  x: -5,
  y: 16,
  abs_x: 5,
  sqrt_y: 4.0,
}
{
  x: -10,
  y: 25,
  abs_x: 10,
  sqrt_y: 5.0,
}

Working with null values

Handle missing or null values gracefully in your data.

Provide default values

Use the else keyword to replace null values:

from {name: "alice", age: 30},
     {name: "bob"},
     {name: "charlie", age: 25}
age = age? else 0
status = status? else "unknown"

{name: "alice", age: 30, status: "unknown"}
{name: "bob", age: 0, status: "unknown"}
{name: "charlie", age: 25, status: "unknown"}

Create new values

Generate new values using built-in functions:

Generate unique identifiers

Use uuid() to create unique identifiers:

from {user: "alice", action: "login"},
     {user: "bob", action: "create"}
event_id = uuid(version="v7")
session_id = uuid()

{
  user: "alice",
  action: "login",
  event_id: "0198147a-d167-7292-80fa-2665c1263279",
  session_id: "a09a7f44-b665-4f95-bc44-c52fbdb8f428",
}
{
  user: "bob",
  action: "create",
  event_id: "0198147a-d167-72ad-80b4-e052c2287add",
  session_id: "030349dc-2585-49ad-af58-d448ff718c05",
}

Generate random numbers

Use random() to generate random values:

from {
  random_float: random(),
  random_int: (random() * 100).int(),
  random_choice: "heads" if random() < 0.5 else "tails",
}

{
  random_float: 0.3215780368890365,
  random_int: 88,
  random_choice: "tails",
}

Access external values

Retrieve values from external sources like the environment, configuration, or files:

Read environment variables

Use env() to access environment variables:

from {
  home_dir: env("HOME"),
  shell: env("SHELL"),
  custom_var: env("MY_APP_CONFIG") else "/default/config",
}

{
  home_dir: "/Users/alice",
  shell: "/opt/homebrew/bin/fish",
  custom_var: "/default/config",
}

Access configuration

Use config() to read Tenzir’s configuration:

from {
  tenzir_config: config(),
}

Read file contents

Use file_contents() to read files:

from {
  api_key: file_contents("/etc/secrets/api_key"),
}

Access secrets

Use secret() to retrieve secrets:

from {
  auth_token: secret("AUTH_TOKEN"),
}

Type inspection

Examine data types at runtime:

Get type information

Use type_of() to inspect value types. Note that this function returns detailed type information as objects:

from {
  str: "hello",
  num: 42,
  float: 3.14,
  bool: true,
  arr: [1, 2, 3],
  obj: {key: "value"}
}
str_type = str.type_of()
num_type = num.type_of()
float_type = float.type_of()
bool_type = bool.type_of()
arr_type = arr.type_of()
obj_type = obj.type_of()

{
  str: "hello",
  num: 42,
  float: 3.14,
  bool: true,
  arr: [1, 2, 3],
  obj: {key: "value"},
  str_type: {name: null, kind: "string", attributes: [], state: null},
  num_type: {name: null, kind: "int64", attributes: [], state: null},
  float_type: {name: null, kind: "double", attributes: [], state: null},
  bool_type: {name: null, kind: "bool", attributes: [], state: null},
  arr_type: {name: null, kind: "list", attributes: [], state: {type: {name: null, kind: "int64", attributes: [], state: null}}},
  obj_type: {name: null, kind: "record", attributes: [], state: {fields: [{name: "key", type: {name: null, kind: "string", attributes: [], state: null}}]}}
}

Get type identifiers

Use type_id() for type comparison:

from {value1: "text", value2: 123, value3: "456"}
type1 = value1.type_id()
type2 = value2.type_id()
type3 = value3.type_id()
same_type = value1.type_id() == value3.type_id()

{
  value1: "text",
  value2: 123,
  value3: "456",
  type1: "2476398993549b5",
  type2: "5b0d4f0b0b167404",
  type3: "2476398993549b5",
  same_type: true,
}

Combining transformations

Real-world data often requires multiple transformations:

from {temp_f: "72.5", location: "  new york  "},
     {temp_f: "89.1", location: "los angeles"}
temp_c = ((temp_f.float() - 32) * 5 / 9).round()
location = location.trim().to_title()
reading = f"{temp_c}°C in {location}"

{
  temp_f: "72.5",
  location: "New York",
  temp_c: 23,
  reading: "23°C in New York",
}
{
  temp_f: "89.1",
  location: "Los Angeles",
  temp_c: 32,
  reading: "32°C in Los Angeles",
}

Best practices

1. Validate before converting: Check that values can be converted to avoid errors.

2. Use appropriate types: Convert to the most specific type needed (e.g., int instead of float for whole numbers).

3. Handle edge cases: Always consider what happens with null values or invalid input.

4. Chain operations efficiently: Combine multiple transformations in a single set statement when possible.

Related guides

• Filter and select data - Learn about filtering and field selection
• Shape data - Overview of all shaping operations
• Manipulate strings - Advanced string manipulation techniques