Transform values

Transforming values is a fundamental part of data processing. This guide shows you how to convert between different data types, perform basic calculations, and manipulate simple values within your events.

Type conversions

TQL provides functions to convert values between different types. This is essential when data arrives in the wrong format or when you need specific types for further processing.

Convert to numbers

Use fnint and fnfloat to convert values to numeric types:

from {price: "42", quantity: "3.5"},
     {price: "99", quantity: "1.0"}
price = price.int()
quantity = quantity.float()

{price: 42, quantity: 3.5}
{price: 99, quantity: 1.0}

Convert to strings

Use fnstring to convert any value to its string representation:

from {status: 200, ratio: 0.95},
     {status: 404, ratio: 0.05}
message = status.string() + " - " + (ratio * 100).string() + "%"

{status: 200, ratio: 0.95, message: "200 - 95.0%"}
{status: 404, ratio: 0.05, message: "404 - 5.0%"}

Parse times and durations

Convert strings to time values with fntime:

from {timestamp: "2024-01-15"},
     {timestamp: "2024-02-20"}
parsed_time = timestamp.time()

{timestamp: "2024-01-15", parsed_time: 2024-01-15T00:00:00Z}
{timestamp: "2024-02-20", parsed_time: 2024-02-20T00:00:00Z}

Convert strings to durations with fnduration:

from {interval: "5s"},
     {interval: "2.5min"}
parsed_duration = interval.duration()

{interval: "5s", parsed_duration: 5s}
{interval: "2.5min", parsed_duration: 2.5min}

Convert to unsigned integers

Use fnuint for non-negative integers:

from {count: "42", ratio: 3.7},
     {count: "-5", ratio: 2.3}
count_uint = count.uint()
ratio_uint = ratio.uint()

{count: "42", ratio: 3.7, count_uint: 42, ratio_uint: 3}
{count: "-5", ratio: 2.3, count_uint: null, ratio_uint: 2}

This pipeline elicits the following warning:

warning: `uint` failed to convert some string
 --> /tmp/pipeline.tql:3:14
  |
3 | count_uint = count.uint()
  |              ~~~~~
  |

Work with IP addresses and subnets

TQL supports IP address and subnet literals directly. You can also parse them from strings using fnip and fnsubnet:

from {direct_ip: 192.168.1.1, direct_subnet: 10.0.0.0/24},
     {direct_ip: ::1, direct_subnet: 2001:db8::/32}
ipv6_check = direct_ip.is_v6()

{
  direct_ip: 192.168.1.1,
  direct_subnet: 10.0.0.0/24,
  ipv6_check: false,
}
{
  direct_ip: ::1,
  direct_subnet: 2001:db8::/32,
  ipv6_check: true,
}

Parse from strings when needed:

from {client: "192.168.1.1", network: "10.0.0.0/24"},
     {client: "10.0.0.5", network: "192.168.0.0/16"}
client_ip = client.ip()
network_subnet = network.subnet()

{
  client: "192.168.1.1",
  network: "10.0.0.0/24",
  client_ip: 192.168.1.1,
  network_subnet: 10.0.0.0/24,
}
{
  client: "10.0.0.5",
  network: "192.168.0.0/16",
  client_ip: 10.0.0.5,
  network_subnet: 192.168.0.0/16,
}

IP address inspection

Analyze and categorize IP addresses with inspection functions:

Check IP address types

Use IP inspection functions like fnis_v4, fnis_v6, fnis_private, fnis_global, fnis_loopback, and fnis_multicast to analyze addresses:

from {ip1: 192.168.1.1, ip2: 8.8.8.8, ip3: ::1},
     {ip1: 10.0.0.1, ip2: 224.0.0.1, ip3: 2001:db8::1}
is_v4 = ip1.is_v4()
is_v6 = ip3.is_v6()
is_private = ip1.is_private()
is_global = ip2.is_global()
is_loopback = ip3.is_loopback()
is_multicast = ip2.is_multicast()

{
  ip1: 192.168.1.1,
  ip2: 8.8.8.8,
  ip3: ::1,
  is_v4: true,
  is_v6: true,
  is_private: true,
  is_global: true,
  is_loopback: true,
  is_multicast: false,
}
{
  ip1: 10.0.0.1,
  ip2: 224.0.0.1,
  ip3: 2001:db8::1,
  is_v4: true,
  is_v6: true,
  is_private: true,
  is_global: false,
  is_loopback: false,
  is_multicast: true,
}

Categorize IP addresses

Get detailed IP address classification with fnip_category:

from {client: "192.168.1.100", server: "8.8.8.8", local: "127.0.0.1"},
     {client: "10.0.0.5", server: "224.0.0.251", local: "::1"}
client_category = client.ip().ip_category()
server_category = server.ip().ip_category()
local_category = local.ip().ip_category()

{
  client: "192.168.1.100",
  server: "8.8.8.8",
  local: "127.0.0.1",
  client_category: "private",
  server_category: "global",
  local_category: "loopback",
}
{
  client: "10.0.0.5",
  server: "224.0.0.251",
  local: "::1",
  client_category: "private",
  server_category: "multicast",
  local_category: "loopback",
}

Check link-local addresses

Identify link-local addresses with fnis_link_local:

from {addr1: 169.254.1.1, addr2: fe80::1, addr3: 192.168.1.1},
     {addr1: 169.254.0.1, addr2: 2001:db8::1, addr3: 10.0.0.1}
link_local1 = addr1.is_link_local()
link_local2 = addr2.is_link_local()
link_local3 = addr3.is_link_local()

{
  addr1: 169.254.1.1,
  addr2: fe80::1,
  addr3: 192.168.1.1,
  link_local1: true,
  link_local2: true,
  link_local3: false,
}
{
  addr1: 169.254.0.1,
  addr2: 2001:db8::1,
  addr3: 10.0.0.1,
  link_local1: true,
  link_local2: false,
  link_local3: false,
}

Basic string operations

Transform strings with simple operations to clean and standardize your data.

Change case

Convert strings to different cases:

from {name: "alice smith", code: "xyz"},
     {name: "BOB JONES", code: "ABC"}
name = name.to_title()
code = code.to_upper()

{name: "Alice Smith", code: "XYZ"}
{name: "Bob Jones", code: "ABC"}

Trim whitespace

Remove unwanted whitespace from strings:

from {input: "  hello  ", data: "world   "},
     {input: "   test", data: "  value  "}
input = input.trim()
data = data.trim()

{input: "hello", data: "world"}
{input: "test", data: "value"}

Capitalize strings

Capitalize the first letter of a string:

from {word: "hello", phrase: "good morning"},
     {word: "world", phrase: "how are you"}
word = word.capitalize()

{word: "Hello", phrase: "good morning"}
{word: "World", phrase: "how are you"}

Mathematical operations

Perform calculations on numeric values within your events.

Basic arithmetic

Use standard arithmetic operators:

from {a: 10, b: 3},
     {a: 20, b: 4}
sum = a + b
diff = a - b
product = a * b
quotient = (a / b).int()

{a: 10, b: 3, sum: 13, diff: 7, product: 30, quotient: 3}
{a: 20, b: 4, sum: 24, diff: 16, product: 80, quotient: 5}

Rounding numbers

Round numbers to specific precision:

from {value: 3.14159},
     {value: 2.71828}
rounded = value.round()
ceil_val = value.ceil()
floor_val = value.floor()

{value: 3.14159, rounded: 3, ceil_val: 4, floor_val: 3}
{value: 2.71828, rounded: 3, ceil_val: 3, floor_val: 2}

Mathematical functions

Use fnabs for absolute values and fnsqrt for square roots:

from {x: -5, y: 16},
     {x: -10, y: 25}
abs_x = x.abs()
sqrt_y = y.sqrt()

{
  x: -5,
  y: 16,
  abs_x: 5,
  sqrt_y: 4.0,
}
{
  x: -10,
  y: 25,
  abs_x: 10,
  sqrt_y: 5.0,
}

Working with null values

Handle missing or null values gracefully in your data.

Provide default values

Use the else keyword to replace null values:

from {name: "alice", age: 30},
     {name: "bob"},
     {name: "charlie", age: 25}
age = age? else 0
status = status? else "unknown"

{name: "alice", age: 30, status: "unknown"}
{name: "bob", age: 0, status: "unknown"}
{name: "charlie", age: 25, status: "unknown"}

Normalize sentinel values

Many data sources use string placeholders like "None", "N/A", "-", or empty strings to represent missing values. Use replace to normalize these to actual null:

from {status: "active", error: "None"},
     {status: "N/A", error: "timeout"},
     {status: "-", error: ""}
replace what="None", with=null
replace what="N/A", with=null
replace what="-", with=null
replace what="", with=null

{status: "active", error: null}
{status: null, error: "timeout"}
{status: null, error: null}

To normalize a specific field rather than all matching strings, specify the field name:

from {enabled: "YES", disabled: "NO"}
replace enabled, what="YES", with=true
replace enabled, what="NO", with=false
replace disabled, what="YES", with=true
replace disabled, what="NO", with=false

{enabled: true, disabled: false}

Create new values

Generate new values using built-in functions:

Generate unique identifiers

Use fnuuid to create unique identifiers:

from {user: "alice", action: "login"},
     {user: "bob", action: "create"}
event_id = uuid(version="v7")
session_id = uuid()

{
  user: "alice",
  action: "login",
  event_id: "0198147a-d167-7292-80fa-2665c1263279",
  session_id: "a09a7f44-b665-4f95-bc44-c52fbdb8f428",
}
{
  user: "bob",
  action: "create",
  event_id: "0198147a-d167-72ad-80b4-e052c2287add",
  session_id: "030349dc-2585-49ad-af58-d448ff718c05",
}

Generate random numbers

Use fnrandom to generate random values:

from {
  random_float: random(),
  random_int: (random() * 100).int(),
  random_choice: "heads" if random() < 0.5 else "tails",
}

{
  random_float: 0.3215780368890365,
  random_int: 88,
  random_choice: "tails",
}

Access external values

Retrieve values from external sources like the environment, configuration, or files:

Read environment variables

Use fnenv to access environment variables:

from {
  home_dir: env("HOME"),
  shell: env("SHELL"),
  custom_var: env("MY_APP_CONFIG") else "/default/config",
}

{
  home_dir: "/Users/alice",
  shell: "/opt/homebrew/bin/fish",
  custom_var: "/default/config",
}

Access configuration

Use fnconfig to read Tenzir’s configuration:

from {
  tenzir_config: config(),
}

Read file contents

Use fnfile_contents to read files:

from {
  api_key: file_contents("/etc/secrets/api_key"),
}

Access secrets

Use fnsecret to retrieve secrets:

from {
  auth_token: secret("AUTH_TOKEN"),
}

Type inspection

Examine data types at runtime:

Get type information

Use fntype_of to inspect value types. Note that this function returns detailed type information as objects:

from {
  str: "hello",
  num: 42,
  float: 3.14,
  bool: true,
  arr: [1, 2, 3],
  obj: {key: "value"}
}
str_type = str.type_of()
num_type = num.type_of()
float_type = float.type_of()
bool_type = bool.type_of()
arr_type = arr.type_of()
obj_type = obj.type_of()

{
  str: "hello",
  num: 42,
  float: 3.14,
  bool: true,
  arr: [1, 2, 3],
  obj: {key: "value"},
  str_type: {name: null, kind: "string", attributes: [], state: null},
  num_type: {name: null, kind: "int64", attributes: [], state: null},
  float_type: {name: null, kind: "double", attributes: [], state: null},
  bool_type: {name: null, kind: "bool", attributes: [], state: null},
  arr_type: {name: null, kind: "list", attributes: [], state: {type: {name: null, kind: "int64", attributes: [], state: null}}},
  obj_type: {name: null, kind: "record", attributes: [], state: {fields: [{name: "key", type: {name: null, kind: "string", attributes: [], state: null}}]}}
}

Get type identifiers

Use fntype_id for type comparison:

from {value1: "text", value2: 123, value3: "456"}
type1 = value1.type_id()
type2 = value2.type_id()
type3 = value3.type_id()
same_type = value1.type_id() == value3.type_id()

{
  value1: "text",
  value2: 123,
  value3: "456",
  type1: "2476398993549b5",
  type2: "5b0d4f0b0b167404",
  type3: "2476398993549b5",
  same_type: true,
}

Combining transformations

Real-world data often requires multiple transformations:

from {temp_f: "72.5", location: "  new york  "},
     {temp_f: "89.1", location: "los angeles"}
temp_c = ((temp_f.float() - 32) * 5 / 9).round()
location = location.trim().to_title()
reading = f"{temp_c}°C in {location}"

{
  temp_f: "72.5",
  location: "New York",
  temp_c: 23,
  reading: "23°C in New York",
}
{
  temp_f: "89.1",
  location: "Los Angeles",
  temp_c: 32,
  reading: "32°C in Los Angeles",
}

Best practices

1. Validate before converting: Check that values can be converted to avoid errors.

2. Use appropriate types: Convert to the most specific type needed (e.g., int instead of float for whole numbers).

3. Handle edge cases: Always consider what happens with null values or invalid input.

4. Chain operations efficiently: Combine multiple transformations in a single set statement when possible.

See also

• Filter and select data
• Manipulate strings
• Clean up values