This section explains the internal architecture of Threat Bus, the missing link to connect open-source security tools. By security tools we mean for example network intrusion detection systems (NIDS) like Zeek, telemetry engines like VAST, or intelligence platforms like MISP.
Threat Bus is a plugin-based application. It relies on pluggy, the "minimalist production ready plugin system". Almost all functionality is implemented in either backbone or application plugins. Please refer to the plugins documentation page for more details. This page focuses on Threat Bus internal logic, like subscription management and plugin life cycle.
Threat Bus is responsible for launching and initializing all installed plugins with the user-provided configuration. It provides some rudimentary data structures for message exchange and subscription management, as well as two callbacks for (un)subscribing to the bus. The following graphic shows a high-level architecture overview. Application plugins are colored green, backbone plugins are colored red.
Plugins are not aware of each other. All installations should be managed by
pip. Threat Bus becomes aware of new plugin installations via their
entry_point declaration in their
setup.py. For example, see the
file of the Zeek plugin. The entry point varies for application and backbone
Internally, Threat Bus keeps lists of all registered plugins of each type. This
housekeeping allows to forward requests to all installed plugins at once. For
example, when a new app registers, the responsible application plugin invokes a
callback. Threat Bus then uses its housekeeping list to forward the subscription
to all installed backbone plugins. Another example are
snapshot requests. When a
comes in with a new subscription, Threat Bus uses the list of installed plugins
to forward the request to all known application plugins. Those that implement
the snapshot feature can execute it (e.g., MISP).
Subscriptions and Message Passing
Threat Bus and all plugins are implemented with python threads and thread-safe, synchronized queues. The main loop of Threat Bus should never be blocked. That is why each plugin should start its own thread for communicating with subscribers (i.e., over the wire).
On start-up, Threat Bus creates one global queue for incoming messages. Let's
inq. This queue is passed to all installed plugins -- backbones and
application plugins alike. Application plugins write to the
read from it.
App Subscription Flow
Threat Bus provides two callbacks to all application plugins for subscribing and unsubscribing apps (like e.g., Zeek). The signature of the callbacks looks as follows.
Whenever a new app (e.g., a new Zeek instance) subscribes to the bus, the
responsible plugin creates a new queue. Let's call that
provision incoming messages from the
inq to all subscribers (all the many
outq_ns). But how do backbones become aware of new queues?
This is done via the subscribe callback. With that, application plugins pass the
outqs to Threat Bus. Threat Bus then instructs all registered
backbones to provision messages for the requested topic to the new queue, in
Once subscribed, application plugins read from the
outqs they created. The
plugins are responsible to forward all messages that appear in that
the subscribed app. How that is done, for example over the wire, is
implementation specific logic and handled by the plugin (e.g., via broker to a
Zeek instance or via ZeroMQ to VAST).
Unsubscription works just as subscription, via a callback to Threat Bus. A
subscribed app, e.g., a Zeek instance, unsubscribes at the responsible app
plugin. The plugin passes the topic and
outq of the subscriber to Threat Bus
via the callback displayed above. Threat Bus then instructs all backbones to
forget about the said