One thing we are observing is that organizations are actively seeking out
solutions to better manage their security data operations. Until recently, they
have been aggressively repurposing common data and observability tools. I
believe that this is a stop-gap measure because there was no alternative. But
now there is a growing ecosystem of security data operations tools to support
the modern security data stack. Ross Haleliuk's epic
article
lays this out at length.
In this article I am explaining the underlying design principles for developing
our own data pipeline engine, coming from the perspective of security teams that
are building out their detection and response architecture. These principles
emerged during design and implementation. Many times, we asked ourselves "what's
the right way of solving this problem?" We often went back to the drawing board
and started challenging existing approaches, such as what a data source is, or
what a connector should do. To our surprise, we found a coherent way to answer
these questions without having to make compromises. When things feel Just Right,
it is a good sign to have found the right solution for a particular problem.
What we are describing here are the lessons learned from studying other systems,
distilled as principles to follow for others.
void bytes events bytes events void 1 2 3 4 5 implicit network connection explicit network connection A B A A C B A A A C Operators Format Connector Bytes Parser Printer Loader Saver I/O Events Parsing/Printing Computation Source Remaining Pipeline Tenzir Node S3 Bucket File Fluent Bit
