Sends and receives HTTP/1.1 requests.
from_http url:string, [method=string, payload=string, headers=record, metadata_field=field, paginate=record->string, paginate_delay=duration, connection_timeout=duration, max_retry_count=int, retry_delay=duration, tls=bool, certfile=string, keyfile=string, password=string { … }]from_http url:string, server=true, [metadata_field=field, responses=record, max_request_size=int, tls=bool, certfile=string, keyfile=string, password=string { … }]
Description
Section titled “Description”The from_http
operator issues HTTP requests or spins up an HTTP/1.1 server on
a given address and forwards received requests as events.
url: string
Section titled “url: string”URL to listen on or to connect to.
Must have the form <host>:<port>
when server=true
.
method = string (optional)
Section titled “method = string (optional)”One of the following HTTP method to use when using the client:
get
head
post
put
del
connect
options
trace
Defaults to get
, or post
if payload
is specified.
payload = string (optional)
Section titled “payload = string (optional)”Payload to send with the HTTP request.
headers = record (optional)
Section titled “headers = record (optional)”Record of headers to send with the request.
metadata_field = field (optional)
Section titled “metadata_field = field (optional)”Field to insert metadata into when using the parsing pipeline.
The response metadata (when using the client mode) has the following schema:
Field | Type | Description |
---|---|---|
code | uint64 | The HTTP status code of the response. |
headers | record | The response headers. |
The request metadata (when using the server mode) has the following schema:
Field | Type | Description |
---|---|---|
headers | record | The request headers. |
query | record | The query parameters of the request. |
path | string | The path requested. |
fragment | string | The URI fragment of the request. |
method | string | The HTTP method of the request. |
version | string | The HTTP version of the request. |
paginate = record -> string (optional)
Section titled “paginate = record -> string (optional)”A lambda expression to evaluate against the result of the request (optionally parsed by the given pipeline). If the expression evaluation is successful and non-null, the resulting string is used as the URL for a new GET request with the same headers.
paginate_delay = duration (optional)
Section titled “paginate_delay = duration (optional)”The duration to wait between consecutive paginatation requests.
Defaults to 0s
.
connection_timeout = duration (optional)
Section titled “connection_timeout = duration (optional)”Timeout for the connection.
Defaults to 5s
.
max_retry_count = int (optional)
Section titled “max_retry_count = int (optional)”The maximum times to retry a failed request. Every request has its own retry count.
Defaults to 0
.
retry_delay = duration (optional)
Section titled “retry_delay = duration (optional)”The duration to wait between each retry.
Defaults to 1s
.
server = bool (optional)
Section titled “server = bool (optional)”Whether to spin up an HTTP server or act as an HTTP client.
Defaults to false
, i.e., the HTTP client.
responses = record (optional)
Section titled “responses = record (optional)”Specify custom responses for endpoints on the server. For example,
responses = { "/resource/create": { code: 200, content_type: "text/html", body: "Created!" }, "/resource/delete": { code: 401, content_type: "text/html", body: "Unauthorized!" }}
creates two special routes on the server with different responses.
Requests to an unspecified endpoint are responded with HTTP Status 200 OK
.
max_request_size = int (optional)
Section titled “max_request_size = int (optional)”The maximum size of an incoming request to accept.
Defaults to 10Mib
.
tls = bool (optional)
Section titled “tls = bool (optional)”Enables TLS.
Defaults to false
.
certfile = string (optional)
Section titled “certfile = string (optional)”Path to the client certificate. Required for server if tls
is true
.
keyfile = string (optional)
Section titled “keyfile = string (optional)”Path to the key for the client certificate. Required for server if tls
is true
.
password = string (optional)
Section titled “password = string (optional)”Password for keyfile.
{ … } (optional)
Section titled “{ … } (optional)”A pipeline that receives the response body as bytes, allowing parsing per request. This is especially useful in scenarios where the response body can be parsed into multiple events.
If not provided, the operator will attempt to infer the parsing operator from
the Content-Type
header. Should this inference fail (e.g., unsupported or
missing Content-Type
), the operator raises an error.
Examples
Section titled “Examples”Make a GET request
Section titled “Make a GET request”Make a request to urlscan.io to search for
scans for tenzir.com
and get the first result.
from_http "https://urlscan.io/api/v1/search?q=tenzir.com"unroll resultshead 1
{ results: { submitter: { ... }, task: { ... }, stats: { ... }, page: { ... }, _id: "0196edb1-521e-761f-9d62-1ca4cfad5b30", _score: null, sort: [ "1747744570133", "\"0196edb1-521e-761f-9d62-1ca4cfad5b30\"" ], result: "https://urlscan.io/api/v1/result/0196edb1-521e-761f-9d62-1ca4cfad5b30/", screenshot: "https://urlscan.io/screenshots/0196edb1-521e-761f-9d62-1ca4cfad5b30.png", }, total: 9, took: 296, has_more: false,}
Paginated API Requests
Section titled “Paginated API Requests”Use the paginate
parameter to handle paginated APIs:
from_http "https://api.example.com/data", paginate=(x => x.next_url?)
This sends a GET request to the initial URL and evaluates the x.next_url
field
in the response to determine the next URL for subsequent requests.
Retry Failed Requests
Section titled “Retry Failed Requests”Configure retries for failed requests:
from_http "https://api.example.com/data", max_retry_count=3, retry_delay=2s
This tries up to 3 times, waiting 2 seconds between each retry.
Listen on port 8080
Section titled “Listen on port 8080”Spin up a server with:
from_http "0.0.0.0:8080", server=true, metadata_field=metadata
Send a request to the HTTP endpoint via curl
:
echo '{"key": "value"}' | gzip | curl localhost:8080 --data-binary @- -H 'Content-Encoding: gzip' -H 'Content-Type: application/json'
Observe the request in the Tenzir pipeline, parsed and decompressed:
{ key: "value", metadata: { headers: { Host: "localhost:8080", "User-Agent": "curl/8.13.0", Accept: "*/*", "Content-Encoding": "gzip", "Content-Length": "37", "Content-Type": "application/json", }, path: "/", method: "post", version: "HTTP/1.1", },}