Skip to main content

14 posts tagged with "pipelines"

View All Tags

· 9 min read
Matthias Vallentin

How would you create a contextualization engine? What are the essential building blocks? We asked ourselves these questions after studying what's out there and built from scratch a high-performance contextualization framework in Tenzir. This blog post introduces this brand-new framework, provides usage examples, and describes how you can build your own context plugin.

· 15 min read
Matthias Vallentin

One thing we are observing is that organizations are actively seeking out solutions to better manage their security data operations. Until recently, they have been aggressively repurposing common data and observability tools. I believe that this is a stop-gap measure because there was no alternative. But now there is a growing ecosystem of security data operations tools to support the modern security data stack. Ross Haleliuk's epic article lays this out at length.

In this article I am explaining the underlying design principles for developing our own data pipeline engine, coming from the perspective of security teams that are building out their detection and response architecture. These principles emerged during design and implementation. Many times, we asked ourselves "what's the right way of solving this problem?" We often went back to the drawing board and started challenging existing approaches, such as what a data source is, or what a connector should do. To our surprise, we found a coherent way to answer these questions without having to make compromises. When things feel Just Right, it is a good sign to have found the right solution for a particular problem. What we are describing here are the lessons learned from studying other systems, distilled as principles to follow for others.

· 5 min read
Oliver Rochford

In today's digital age, businesses are under immense pressure to bolster their cybersecurity. Understanding the financial implications of security tools is vital to ensure optimal ROI through risk reduction and breach resilience. This is particularly true for consumption-based security solutions like Security Information and Event Management (SIEM).

· 7 min read
Daniel Kostuj
Matthias Vallentin

We've just released Tenzir v4.2 that introduces two new connectors: S3 and GCS for interacting with blob storage and ZeroMQ for writing distributed multi-hop pipelines. There's also a new lines parser for easier text processing and a bunch of PCAP quality-of-life improvements.

· 4 min read
Dominik Lohmann

After our successful launch of app.tenzir.com of Tenzir v4.0 at Black Hat, the new v4.1 release continues with several enhancements based on early feedback. We bring to you a (i) new mechanism to pause pipelines, (ii) a new operator to match Sigma rules, (iii) new operators for in-pipeline (de)compression, and (iv) a revamp of the show operator.

· 8 min read
Matthias Vallentin

Elastic just released their new pipeline query language called ES|QL. This is a conscious attempt to consolidate the language zoo in the Elastic ecosystem (queryDSL, EQL, KQL, SQL, Painless, Canvas/Timelion). Elastic said that they worked on this effort for over a year. The documentation is still sparse, but we still tried to read between the lines to understand what this new pipeline language has to offer.

· 2 min read
Oliver Rochford

Staying ahead in the realm of cybersecurity means relentlessly navigating an endless sea of emerging threats and ever-increasing data volumes. The battle to stay one step ahead can often feel overwhelming, especially when your organization's data costs are skyrocketing.

· 5 min read
Oliver Rochford

We're overjoyed to announce our highly-anticipated security data pipeline platform at the renowned BlackHat conference in Las Vegas. The launch marks a milestone in our journey to bring simplicity to data engineering for cybersecurity operations, and to bring a cost-efficient way to tackle the increasingly complex data engineering challenges that security teams confront daily.

· 9 min read
Matthias Vallentin

Our Tenzir Query Language (TQL) is a pipeline language that works by chaining operators into data flows. When we designed TQL, we specifically studied Splunk's Search Processing Language (SPL), as it generally leaves a positive impression for security analysts that are not data engineers. Our goal was to take all the good things of SPL, but provide a more powerful language without compromising simplicity. In this blog post, we explain how the two languages differ using concrete threat hunting examples.

· 5 min read
Matthias Vallentin

Did you know that Zeek supports log rotation triggers, so that you can do anything you want with a newly rotated batch of logs?