Overview

This section explains the internal architecture of Threat Bus, the missing link to connect open-source security tools. By security tools we mean for example network intrusion detection systems (NIDS) like Zeek, telemetry engines like VAST, or intelligence platforms like MISP.

Threat Bus is a plugin-based application. It relies on pluggy, the "minimalist production ready plugin system". Almost all functionality is implemented in either backbone or application plugins. Please refer to the plugins documentation page for more details. This page focuses on Threat Bus internal logic, like subscription management and plugin life cycle.

High-Level Structure

Threat Bus is responsible for launching and initializing all installed plugins with the user-provided configuration. It provides some rudimentary data structures for message exchange and subscription management, as well as two callbacks for (un)subscribing to the bus. The following graphic shows a high-level architecture overview. Application plugins are colored green, backbone plugins are colored red.

Plugins are not aware of each other. All installations should be managed by pip. Threat Bus becomes aware of new plugin installations via their entry_point declaration in their setup.py. For example, see the setup.py file of the Zeek plugin. The entry point varies for application and backbone plugins.

Internally, Threat Bus keeps lists of all registered plugins of each type. This housekeeping allows to forward requests to all installed plugins at once. For example, when a new app registers, the responsible application plugin invokes a callback. Threat Bus then uses its housekeeping list to forward the subscription to all installed backbone plugins. Another example are snapshot requests. When a snapshot request comes in with a new subscription, Threat Bus uses the list of installed plugins to forward the request to all known application plugins. Those that implement the snapshot feature can execute it (e.g., MISP).

Subscriptions and Message Passing

Threat Bus and all plugins are implemented with python threads and thread-safe, synchronized queues. The main loop of Threat Bus should never be blocked. That is why each plugin should start its own thread for communicating with subscribers (i.e., over the wire).

On start-up, Threat Bus creates one global queue for incoming messages. Let's call it inq. This queue is passed to all installed plugins -- backbones and application plugins alike. Application plugins write to the inq, backbones read from it.

App Subscription Flow

Threat Bus provides two callbacks to all application plugins for subscribing and unsubscribing apps (like e.g., Zeek). The signature of the callbacks looks as follows.

subscribe(topic, outq, snapshot_time_delta=None)
unsubscribe(topic, outq)

Subscriptions

Whenever a new app (e.g., a new Zeek instance) subscribes to the bus, the responsible plugin creates a new queue. Let's call that outq_1. Backbones provision incoming messages from the inq to all subscribers (all the many outq_ns). But how do backbones become aware of new queues?

This is done via the subscribe callback. With that, application plugins pass the newly created outqs to Threat Bus. Threat Bus then instructs all registered backbones to provision messages for the requested topic to the new queue, in this case outq_1.

Once subscribed, application plugins read from the outqs they created. The plugins are responsible to forward all messages that appear in that outq to the subscribed app. How that is done, for example over the wire, is implementation specific logic and handled by the plugin (e.g., via broker to a Zeek instance or via ZeroMQ to VAST).

Unsubscriptions

Unsubscription works just as subscription, via a callback to Threat Bus. A subscribed app, e.g., a Zeek instance, unsubscribes at the responsible app plugin. The plugin passes the topic and outq of the subscriber to Threat Bus via the callback displayed above. Threat Bus then instructs all backbones to forget about the said outq.