ZeroMQ App Plugin

The ZeroMQ app plugin enables any application that can communicate via ZeroMQ to subscribe to Threat Bus.

The plugin defines a simple protocol for managing (un)subscriptions from apps. In the following, we describe how to install and configure the plugin, and how the management protocol enables building new applications.

Example Application — VAST - Threat Bus Connector

An example application that implements the management protocol is the pyvast-threatbus application. We use this wrapper because VAST cannot communicate with Threat Bus natively. This Python application generates VAST queries for indicators and reports the results back as sightings. It is documented below.

Installation

We recommend to use a virtual environment for the installation. Set it up as follows:

virtualenv venv
source venv/bin/activate

The plugin itself is published as PyPI package. All required dependencies will be installed automatically when installing the plugin.

pip install threatbus-zmq-app

Configuration

The plugin starts three listening ZeroMQ endpoints. The endpoint characteristics for listening can be configured as follows:

...
plugins:
apps:
zmq-app:
host: "127.0.0.1"
manage: 13370 # the port used for management messages
pub: 13371 # the port used to publish messages to connected apps
sub: 13372 # the port used to receive messages from connected apps
...

The manage endpoint handles (un)subscriptions, the pub endpoint publishes new messages to all subscribers, and subscribers use the sub endpoint to report back sightings. Check out the Communication Flow section for details about these endpoints.

Communication Flow

The ZeroMQ app plugin provides three endpoints for subscribers. Applications initially only require the manage endpoint. The plugin sends the details about the pub/sub ports to new subscribers during the subscription handshake.

Threat Bus (or rather, the ZeroMQ app plugin) creates a unique queue and topic for each new subscriber. Upon successful registration, Threat Bus sends the topic name to the subscriber. Subscribers then bind to the pub endpoint using that topic. See the management protocol section for details.

Unique topics make snapshotting possible. Without unique topics, every subscriber would potentially see the requested snapshot data of other subscribers.

Management Protocol

Subscriptions and unsubscriptions are referred to as management messages, which are JSON formatted and exchanged via the manage ZMQ endpoint that the plugin exposes.

The manage endpoint uses the ZeroMQ Request/Reply pattern for message exchange. That means, all messages get an immediate response from the server. With each JSON reply, the zmq-app plugin sends a status field that indicates the success of the requested operation.

Subscription

To subscribe to Threat Bus via the zmq-app plugin, an app needs to send a JSON struct as follows to the manage endpoint of the plugin:

{
"action": "subscribe",
"topic": <TOPIC>, # either 'threatbus/sighting' or 'threatbus/intel'
"snapshot": <SNAPSHOT> # number of days for a snapshot, 0 for no snapshot
}

In response, the app will either receive a success or error response.

  • Error response:

    {
    "status": "error"
    }
  • Success response:

    {
    "topic": <P2P_TOPIC>,
    "pub_endpoint": "127.0.0.1:13371",
    "sub_endpoint": "127.0.0.1:13372",
    "status": "success",
    }

    The pub_endpoint and sub_endpoint of the reply provide the endpoints that an app should connect with to participate in the pub/sub message exchange. The app can access these endpoints following the ZeroMQ Pub/Sub pattern. The plugin will publish messages on the pub_endpoint and accept messages on the sub_endpoint.

    The topic field of the response contains a unique topic for the client. That topic must be used to receive messages. The unique topic is a 32 characters wide random string and guarantees that other subscribers won't accidentally see traffic that should only be visible to the new client. See below for more details on pub/sub via ZeroMQ.

Unsubscription

To unsubscribe, a connected app has to send a JSON struct to the manage endpoint of the plugin, as follows:

{
"action": "unsubscribe",
"topic": <P2P_TOPIC> # the 32-characters random topic that the app received during subscription handshake
}

In response, the app will either receive a success or error response.

  • Error response:
    {
    "status": "error"
    }
  • Success response:
    {
    "status": "success"
    }

Heartbeats

The plugin supports synchronous heartbeats from subscribed apps. Heartbeats allow Threat Bus and the connected apps to mutually ensure that their communication partners are still alive.

Subscribed apps can send heartbeat messages with the following JSON format to the manage endpoint of this plugin:

{
"action": "heartbeat",
"topic": <P2P_TOPIC> # the 32-characters random topic that the app got during subscription handshake
}

As stated in the beginning of this section, the manage endpoint implements the ZeroMQ Request/Reply pattern. Threat Bus answers immediately to each heartbeat request with either a success or error response.

  • Error response:
    {
    "status": "error"
    }
  • Success response:
    {
    "status": "success"
    }

An error response indicates that either Threat Bus has internal erros or that it lost track of the app's subscription. Note: This only happens when Threat Bus is restarted. Apps can then use that information to re-subscribe.

If Threat Bus does not answer a heartbeat message, it is either down or not reachable (e.g., due to network issues). Plugins can use that information to try again later.

Pub/Sub via ZeroMQ

Once an app has subscribed to Threat Bus via using the manage endpoint of the zmq-app plugin, it has a unique, random p2p_topic (see above). Threat Bus (the zmq-app) uses this topic to publish messages to the plugin. Messages can be of the types specified in threatbus.data, i.e., Intel, Sighting, SnapshotRequest, and SnapshotEnvelope.

ZeroMQ uses prefix matching for pub/sub connections. The zmq-app plugin leverages this feature to indicate the type of each sent message to the subscriber. Hence, an app can simply match the topic suffix to determine the message type.

For example, all Intel items will always be sent on the topic p2p_topic + "intel", all SnapshotRequest items on the topic p2p_topic + "snapshotrequest" and so forth.

Application Example: VAST - Threat Bus Connector

Threat Bus is a publish-subscribe broker for threat intelligence. It expects that applications register themselves at the bus. Since VAST cannot do so on its own, this application implements that functionality in the meantime. Check out the source code of the application to see how it implements the zmq-app management protocol. If you have questions, don't hesitate to chat with us on Matrix.

The app provides a thin layer around PyVAST, VAST's Python CLI bindings. It facilitates message exchange between Threat Bus and a VAST instance.

Installation

You can either run the app directly by cloning the repository and invoking the Python file or you can install it via pip. We recommend to install the app in a virtual environment.

virtualenv venv
source venv/bin/activate
python3 -m pip install pyvast-threatbus

Usage

The app has to be provided with the manage endpoint of the zmq-app plugin to subscribe to Threat Bus. For example, an invocation of the app could look as follows:

./venv/bin/pyvast-threatbus --threatbus="localhost:13370" --snapshot=30

The above call tells the app to contact Threat Bus on localhost:13370 and requests a snapshot for 30 days.

Use the help command to list all available options.

./venv/bin/pyvast-threatbus --help