Talking HTTP

This release brings forth improvements to HTTP support in Tenzir, supporting requests as transformations and paginating APIs.

Download the release on GitHub.

Features

Implement http operator

We implemented the http operator that allows making HTTP/1.1 requests to a URL. The operator also allows paginate APIs based on the responses.

By @raxyte in #5188.

Introduce lambda functions

TQL now supports lambda expressions. They are supported in the where and map functions on list, and on the newly added count_if aggregation function. Instead of [1, 2, 3].map(x, x + 1), use [1, 2, 3].map(x => x + 1). This subtle change makes it obvious that the expression is not evaluated on the entire list, but rather on each element individually.

The count_if aggregation function counts the number of elements in a list that satisfy a given predicate. For example, [1, 2, 3].count_if(x => x > 1) returns 2.

By @dominiklohmann in #5150.

Implement from_http client

The from_http operator now supports HTTP client functionality. This allows sending HTTP/1.1 requests, including support for custom methods, headers, payloads, pagination, retries, and connection timeouts. The operator can be used to fetch data from HTTP APIs and ingest it directly into pipelines.

Make a simple GET request auto-selecting the parser:

from_http "https://api.example.com/data"

Post data to some API:

from_http "https://api.example.com/submit", payload={foo: "bar"}.print_json(),
          headers={"Content-Type": "application/json"}

Paginating APIs:

from_http "https://api.example.com/items",
          paginate=(x => x.next_url if x.has_more? == true)

By @raxyte in #5177.

Changes

load_http deprecated

The from operator now dispatches to from_http for http[s] URLs.

The load_http operator is now deprecated in favor of from_http.

By @raxyte in #5177.

Bug Fixes

Fix a segfault in save_amqp on connection loss

We fixed a crash in save_amqp when trying to send a message after the connection was lost.

By @IyeOnline in #5226.

Fix overflow warning for -9223372036854775808

The lowest 64-bit integer, -9223372036854775808, no longer causes an overflow warning.

By @jachris in #5223.

Fix a crash in to_clickhouse

We fixed an issue when trying to send lists in to_clickhouse that would cause the ClickHouse server to drop the connection.

By @IyeOnline in #5231.

Fix evaluation of null if true else …

The expression null if true else 42 previously returned 42. It now correctly returns null.

By @dominiklohmann in #5150.