MISP Plugin

The Threat Bus MISP plugin enables communication with the MISP Open Source Threat Intelligence Platform. The plugin handles all communication with MISP. It uses either ZeroMQ or Kafka for receiving new intelligence data and reports back sightings to MISP via REST API calls.

MISP Module

The Threat Bus MISP plugin in its current form violates the pub/sub architecture of Threat Bus. That is because the plugin subscribes a listener to ZeroMQ / Kafka, rather than having MISP subscribe itself to Threat Bus. That will be addressed with a MISP module in the future.

The plugin makes the following deployment assumptions:

  1. Indicators of compromise (IoCs) are stored in MISP as attributes.
  2. Either ZeroMQ or Kafka is enabled in MISP for attribute publishing.
  3. The MISP REST API is enabled and an API key is available.

With the help of this plugin all MISP attributes that are published via the web interface are parsed and published on the threatbus/intel topic. Other apps, like Zeek and VAST can consume those intel items by subscribing to that topic.

Vice versa, the MISP plugin subscribes to the threatbus/sightings topic and reports back all IoC matches to the MISP platform via PyMISP.

Installation

The plugin communicates either via ZeroMQ or Kafka. When using Kafka, you have to install librdkafka for the host system that is running Threat Bus. See the prerequisites section of the confluent-kafka python client for more details.

Once the prerequisites are met, install the MISP plugin via pip.

pip install threatbus-misp

Configuration

The plugin can either use ZeroMQ or Kafka to retrieve intelligence items from MISP. It uses the MISP REST API to report back sightings of indicators.

ZeroMQ and Kafka are mutually exclusive, such that Threat Bus does not receive all attribute updates twice. See below for an example configuration.

...
plugins:
misp:
api:
host: https://localhost
ssl: false
key: <MISP_API_KEY>
zmq:
host: localhost
port: 50000
#kafka:
# topics:
# - misp_attribute
# poll_interval: 1.0
# # All config entries are passed as-is to librdkafka
# # https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
# config:
# bootstrap.servers: "localhost:9092"
# group.id: "threatbus"
# auto.offset.reset: "earliest"
...
Kafka Fine-tuning

The MISP plugin forwards all settings from the kafka.config section of the configuration file directly to the Kafka client. The used python consumer is confluent-kafka.Consumer. For a list of all config options please see the official Kafka Client Configuration docs.

Development Setup

Testing the MISP plugin requires a fair lot of third-party components to work correctly with each other. At times, it can be exhausting to research all required steps to setup a fresh testing environment manually. The following guides describe how to set up local, dockerized instances of MISP and Kafka for testing the Threat Bus MISP plugin.

Dockerized Kafka

For a simple, working Kafka Docker setup use the single node example from confluentinc/cp-docker-images.

Store the docker-compose.yaml and modify the Kafka environment variables such that the Docker host (e.g., 172.17.0.1) of your Docker machine is advertised as Kafka listener:

zookeeper:
...
kafka:
...
environment:
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://172.17.0.1:9092 # <-- That is the IP of your Docker host
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
...

For details about Kafka listeners, check out this article. Then start the compose setup via docker-compose up -d.

To test the setup, use the tests/utils/kafka_receiver.py and tests/utils/kafka_sender.py scripts from the Threat Bus repository.

Dockerized MISP

Use DCSO's dockerized MISP to set up a local testing environment, as follows.

Setup a MISP Docker cluster

git clone git@github.com:DCSO/MISP-dockerized.git
cd MISP-dockerized
make install
# follow the dialog...

Edit the docker-compose.yaml

cd current
vim docker-compose.yaml

Find the section misp-server in the configuration and add the following:

misp-server:
...
ports:
- "50000:50000"
...

Restart MISP to accept the new port

make deploy

Enable the Kafka plugin in the MISP web-view

  • Visit https://localhost:80
  • login with your configured credentials
  • Go to Administration -> Server Settings & Maintenance -> Plugin settings Tab
  • Set the following entries
    • Plugin.Kafka_enable -> true
    • Plugin.Kafka_brokers -> 172.17.0.1:9092 <- In this example, 172.17.0.1 is the Docker host, reachable from other Docker networks. The port is reachable when the Kafka Docker setup binds to it globally.
    • Plugin.Kafka_attribute_notifications_enable -> true
    • Plugin.Kafka_attribute_notifications_topic -> misp_attribute <- The topic goes into the threatbus config.yaml

Install Kafka inside the misp-server container

docker exec -ti misp-server bash # pop interactive shell inside the container
apt-get install software-properties-common
apt-get update
# enable stretch-backports to get a recent librdkafka version
add-apt-repository "deb http://deb.debian.org/debian stretch-backports main contrib non-free"
apt-get update
apt-get install librdkafka-dev/stretch-backports
# see https://misp.github.io/MISP/INSTALL.ubuntu1804/#misp-has-a-feature-for-publishing-events-to-kafka-to-enable-it-simply-run-the-following-commands
pecl channel-update pecl.php.net
pecl install rdkafka
echo "extension=rdkafka.so" | tee /etc/php/7.0/mods-available/rdkafka.ini
phpenmod rdkafka
service apache2 restart
exit # leave the Docker container shell

Enable the ZMQ plugin in the MISP web-view

  • Visit https://localhost:80
  • login with your configured credentials
  • Go to Administration -> Server Settings & Maintenance -> Diagnostics Tab
  • Find the ZeroMQ plugin section and enable it
  • Go to Administration -> Server Settings & Maintenance -> Plugin settings Tab
  • Set the entry Plugin.ZeroMQ_attribute_notifications_enable to true

Restart all MISP services

make restart-all