🚀 Features
Section titled “🚀 Features”Custom quotes and doubled quote escaping
Section titled “Custom quotes and doubled quote escaping”Jan 7, 2025 · @IyeOnline · #4837
The read_csv, read_kv, read_ssv, read_tsv and read_xsv operators now
support custom quote characters.
The read_csv, read_ssv, read_tsv and read_xsv operators support doubled
quote escaping.
The read_csv, read_ssv, read_tsv and read_xsv operators now accept
multi-character strings as separators.
The list_sep option for the read_csv, read_ssv, read_tsv and read_xsv
operators can be set to an empty string, which will disable list parsing.
The new string.parse_leef() function can be used to parse a string as a LEEF
message.
Implement float(number|string)
Section titled “Implement float(number|string)”Numbers and string expressions containing numbers can now be converted into
float type values using the float function.
Add user-defined operators to TQL2
Section titled “Add user-defined operators to TQL2”Jan 6, 2025 · @jachris · #4884
User-defined operators can now be written and used in TQL2. To use TQL2, start
your definition with the comment // tql2, or use the --tql2 flag to opt into
TQL2 as the default.
Add duration(string) -> duration
Section titled “Add duration(string) -> duration”Dec 24, 2024 · @raxyte · #4877
The new duration function now allows to parse expressions
resulting in strings as duration values.
Add Snowflake sink
Section titled “Add Snowflake sink”Dec 23, 2024 · @IyeOnline · #4589
We have added a new to_snowflake sink operator, writing events into a
snowflake table.
Add to_opensearch and to_elasticsearch sink operators
Section titled “Add to_opensearch and to_elasticsearch sink operators”Dec 20, 2024 · @raxyte · #4871
A new operator to_opensearch is now available for sending data to
OpenSearch-compatible Bulk API providers including ElasticSearch.
Implement TQL2 from and to
Section titled “Implement TQL2 from and to”Dec 18, 2024 · @IyeOnline · #4805
We have added the from operator that allows you to easily onboard data from
most sources.
For example, you can now write from "https://example.com/file.json.gz"
to automatically deduce the load operator, compression, and format.
We have added the to operator that allows you to easily send data to most
destinations.
For example, you can now write to "ftps://example.com/file.json.gz"
to automatically deduce the save operator, compression, and format.
You can use the new subnet(string) function to parse strings as subnets.
Enhance HTTP connector controls
Section titled “Enhance HTTP connector controls”Several new options are now available for the load_http operator: data,
json, form, skip_peer_verification, skip_hostname_verification,
chunked, and multipart. The skip_peer_verification and
skip_hostname_verification options are now also available for the save_http
operator.
Implement context::erase
Section titled “Implement context::erase”Dec 17, 2024 · @dominiklohmann · #4864
The context::erase operator allows you to selectively remove entries from
contexts.
Port deduplicate to TQL2
Section titled “Port deduplicate to TQL2”Dec 13, 2024 · @dominiklohmann · #4850
The deduplicate operator in TQL2 to help you remove events with a common key.
The operator provides more flexibility than its TQL1 pendant by letting the
common key use any expression, not just a field name. You can also control
timeouts with finer granularity.
Introduce a TQL2-only mode
Section titled “Introduce a TQL2-only mode”Dec 12, 2024 · @dominiklohmann · #4840
Start your Tenzir Node with tenzir-node --tql2 or set the TENZIR_TQL2=true
environment variable to enable TQL2-only mode for your node. In this mode, all
pipelines will run as TQL2, with the old TQL1 pipelines only being available
through the legacy operator. In Q1 2025, this option will be enabled by
default, and later in 2025 the legacy operator and TQL1 support will be
removed entirely.
save_email cleanup
Section titled “save_email cleanup”Dec 11, 2024 · @raxyte · #4848
The save_email now accepts a tls option to specify TLS usage when establishing
the SMTP connection.
Parse x not in y as not x in y
Section titled “Parse x not in y as not x in y”Dec 10, 2024 · @raxyte · #4844
TQL2 now allows writing x not in y as an equivalent to not (x in y) for
better readability.
Implement ip in subnet and subnet in subnet
Section titled “Implement ip in subnet and subnet in subnet”Dec 6, 2024 · @jachris · #4841
Whether an IP address is contained in a subnet can now be checked using
expressions such as 1.2.3.4 in 1.2.0.0/16. Similarly, to check whether a
subnet is included in another subnet, use 1.2.0.0/16 in 1.0.0.0/8.
🔧 Changes
Section titled “🔧 Changes”Split compress/decompress into separate operators
Section titled “Split compress/decompress into separate operators”Jan 6, 2025 · @IyeOnline · #4876
The compress and decompress operators have been deprecated in favor of
separate operators for each compression algorithm. These new operators expose
additional options, such as compress_gzip level=10, format="deflate".
Implement TQL2 from and to
Section titled “Implement TQL2 from and to”Dec 18, 2024 · @IyeOnline · #4805
The topic argument for load_kafka and save_kafka is now a positional
argument, instead of a named argument.
The array version of from that allowed you to create multiple events has been
removed. Instead, you can just pass multiple records to from now.
Make the expression evaluator support heterogeneous results
Section titled “Make the expression evaluator support heterogeneous results”Dec 14, 2024 · @jachris · #4839
Functions can now return values of different types for the same input types. For
example, x.otherwise(y) no longer requires that x has the same type as y.
🐞 Bug Fixes
Section titled “🐞 Bug Fixes”Fix operator parenthesis continuation
Section titled “Fix operator parenthesis continuation”Jan 6, 2025 · @jachris · #4885
Operator invocations that directly use parenthesis but continue after the
closing parenthesis are no longer rejected. For example, where (x or y) and z
is now being parsed correctly.
Fix handling of empty records in write_parquet
Section titled “Fix handling of empty records in write_parquet”Dec 20, 2024 · @dominiklohmann · #4874
write_parquet now gracefully handles nested empty records by replacing them
with nulls. The Apache Parquet format does fundamentally not support empty
nested records.
Fix pipeline manager discarding parse-time warnings
Section titled “Fix pipeline manager discarding parse-time warnings”Dec 17, 2024 · @jachris · #4867
Warnings that happen very early during pipeline startup now correctly show up in the Tenzir Platform.
Make the expression evaluator support heterogeneous results
Section titled “Make the expression evaluator support heterogeneous results”Dec 14, 2024 · @jachris · #4839
Metadata such as @name can now be set to a dynamically computed value that
does not have to be a constant. For example, if the field event_name should be
used as the event name, @name = event_name now correctly assigns the events
their name instead of using the first value.
Validate legacy expressions when splitting for predicate pushdown
Section titled “Validate legacy expressions when splitting for predicate pushdown”Dec 12, 2024 · @jachris · #4861
Pipelines that begin with export | where followed by an expression that does
not depend on the incoming events, such as export | where 1 == 1, no longer
cause an internal error.
save_email cleanup
Section titled “save_email cleanup”Dec 11, 2024 · @raxyte · #4848
The endpoint argument of the save_email operator was documented as optional
but was not parsed as so. This has been fixed and the argument is now
correctly optional.