14 posts tagged with "pipelines"

How would you create a contextualization engine? What are the essential building blocks? We asked ourselves these questions after studying what's out there and built from scratch a high-performance contextualization framework in Tenzir. This blog post introduces this brand-new framework, provides usage examples, and describes how you can build your own context plugin.

One thing we are observing is that organizations are actively seeking out solutions to better manage their security data operations. Until recently, they have been aggressively repurposing common data and observability tools. I believe that this is a stop-gap measure because there was no alternative. But now there is a growing ecosystem of security data operations tools to support the modern security data stack. Ross Haleliuk's epic article lays this out at length.

In this article I am explaining the underlying design principles for developing our own data pipeline engine, coming from the perspective of security teams that are building out their detection and response architecture. These principles emerged during design and implementation. Many times, we asked ourselves "what's the right way of solving this problem?" We often went back to the drawing board and started challenging existing approaches, such as what a data source is, or what a connector should do. To our surprise, we found a coherent way to answer these questions without having to make compromises. When things feel Just Right, it is a good sign to have found the right solution for a particular problem. What we are describing here are the lessons learned from studying other systems, distilled as principles to follow for others.

In today's digital age, businesses are under immense pressure to bolster their cybersecurity. Understanding the financial implications of security tools is vital to ensure optimal ROI through risk reduction and breach resilience. This is particularly true for consumption-based security solutions like Security Information and Event Management (SIEM).

We've just released Tenzir v4.2 that introduces two new connectors: S3 and GCS for interacting with blob storage and ZeroMQ for writing distributed multi-hop pipelines. There's also a new lines parser for easier text processing and a bunch of PCAP quality-of-life improvements.

After our successful launch of app.tenzir.com of Tenzir v4.0 at Black Hat, the new v4.1 release continues with several enhancements based on early feedback. We bring to you a (i) new mechanism to pause pipelines, (ii) a new operator to match Sigma rules, (iii) new operators for in-pipeline (de)compression, and (iv) a revamp of the show operator.

Elastic just released their new pipeline query language called ES|QL. This is a conscious attempt to consolidate the language zoo in the Elastic ecosystem (queryDSL, EQL, KQL, SQL, Painless, Canvas/Timelion). Elastic said that they worked on this effort for over a year. The documentation is still sparse, but we still tried to read between the lines to understand what this new pipeline language has to offer.

Staying ahead in the realm of cybersecurity means relentlessly navigating an endless sea of emerging threats and ever-increasing data volumes. The battle to stay one step ahead can often feel overwhelming, especially when your organization's data costs are skyrocketing.

We're overjoyed to announce our highly-anticipated security data pipeline platform at the renowned BlackHat conference in Las Vegas. The launch marks a milestone in our journey to bring simplicity to data engineering for cybersecurity operations, and to bring a cost-efficient way to tackle the increasingly complex data engineering challenges that security teams confront daily.

Our Tenzir Query Language (TQL) is a pipeline language that works by chaining operators into data flows. When we designed TQL, we specifically studied Splunk's Search Processing Language (SPL), as it generally leaves a positive impression for security analysts that are not data engineers. Our goal was to take all the good things of SPL, but provide a more powerful language without compromising simplicity. In this blog post, we explain how the two languages differ using concrete threat hunting examples.

Did you know that Zeek supports log rotation triggers, so that you can do anything you want with a newly rotated batch of logs?

Zeek turns packets into structured logs. By default, Zeek generates one file per log type and per rotation timeframe. If you don't want to wrangle files and directly process the output, this short blog post is for you.

VAST v3.1 is out. This is a small checkpointing release that brings a few new changes and fixes.

VAST v3.0 is out. This release brings some major updates to the the VAST language, making it easy to write down dataflow pipelines that filter, reshape, aggregate, and enrich security event data. Think of VAST as security data pipelines plus open storage engine.

We released VAST v2.2 🙌! Transforms now have a new name: pipelines. The summarize operator also underwent a facelift, making aggregation functions pluggable and allowing for assigning names to output fields.