Plugins

Experimental Feature

This is an experimental feature: the API is subject to change and robustness is not yet comparable to production-grade features.

VAST has a plugin mechanism that makes it easy to hook into various places of the data processing pipeline and add custom extensions in a safe and sustainable fashion. A set of customization points allow anyone to add new CLI commands, receive a copy of the input stream, spawn queries, or implement integrations with third-party libraries.

Implementation Status

The light-yellow, dashed plugins in the above figure have not yet been implemented. Specifically, the missing plugin types are:

Carrier: a data transport mechanism, like UDP or TCP
Reader: an parser to transform a specific format into VAST data
Writer: a printer to render VAST data in a specific format
Import: a filter applied after having parsed the input at a source
Export: a filter applied before passing query results to a sink

Usage

Installating a Plugin

There exist two ways to deploy plugins:

Static: compiled into VAST. Native plugins part of the VAST code base ship in this form.
Dynamic: loaded as separate shared library, typically used for third-party or binary-only plugins.

Static plugins do not require loading since they are compiled into VAST. Dynamic plugins are shared libraries and therefore must be loaded first into the running VAST process. At startup, VAST looks into the configured vast.plugin-dirs directories in the vast.yaml file to find the configured vast.plugins files.

vast:
  plugin-dirs:
    - .
    - /opt/foo/lib
  plugins:
    - libexample
    - /opt/foo/lib/libexample.so

Before executing functionality, VAST loads the specified plugins via dlopen(3) and attempts to initialize them as plugins. If the version check passes, VAST initializes the plugin with the respective configuration in the vast.yaml file:

plugins:
  example:
    option: 42

After initialization with the configuration options, the plugin is fully operational and VAST will call its functions at the plugin-specific customization points.

Listing All Plugins

You can get a list of all plugins and their respective version by running vast version:

{
  "VAST": "2020.12.16-97-gced115d91-dirty",
  "CAF": "0.17.6",
  "Apache Arrow": "2.0.0",
  "PCAP": "libpcap version 1.9.1",
  "jemalloc": null,
  "plugins": {
    "example": "0.1.0-0"
  }
}

Design

VAST's plugin design follows a few principles:

Complementary: plugins must be additive, no mutation of existing functionality.
Composable: a single plugin can offer multiple features, e.g., add a custom CLI command while simultaneously hooking into the import stream.
Configurable: users should be able to configure the different tuning knobs of a plugin in the vast.yaml.

We found that a class-based plugin hierarchy with name-based registration achieves these design goals, as long as the customization points act orthogonal to each other.

Developing a Plugin

Implementing a new plugin requires the following steps:

Setup the scaffolding
Choose a plugin type
Implement the plugin interface
Process configuration options
Compile the source code
Add unit and integration tests
Package it

Next, we'll discuss each step in more detail.

The Scaffolding

The scaffolding of a plugin includes the CMake glue that makes it possible to use as static or dynamic plugin.

VAST ships with an example plugin that showcases how a typical scaffold looks like. Have a look at the the example plugins directory.

Choosing a Plugin Type

VAST offers a variety of customization points, each of which defines its own API by inheriting from the plugin base class vast::plugin. When writing a new plugin, you can choose a subset of available types by inheriting from the respective plugin classes.

Dreaded Diamond

To avoid common issues with multiple inheritance, all intermediate plugin classes that inherit from vast::plugin use virtual inheritance to avoid issues with the dreaded diamond. When composing multiple plugin types, however, you should use non-virtual public inheritance only.

Please consult the example plugin for a concrete code sample.

Command Plugin

A command plugin adds a new command to the vast executable, at a configurable location in the command hierarchy.

The base class vast::command_plugin defines a factory function make_command() that returns a new command. Concretely, the return value of this function is a std::pair<std::unique_ptr<command>, command::factory>. The first component is the command instance, and the second defines the mapping from command name to command implementation.

Analyzer Plugin

The analyzer plugin hooks into the processing path of data by spawning a new actor inside the server that receives the full stream of table slices.

The base class vast::analyzer_plugin has a typed actor interface system::analyzer_plugin_actor that users must return by overriding the pure virtual factory make_analyzer(vast::system::node_actor::pointer). The analyzer_plugin_actor has the following type:

// See libvast/vast/system/actors.hpp
/// The interface of an ANALYZER PLUGIN actors.
using analyzer_plugin_actor = caf::typed_actor<>
  // Conform to the protocol of the STREAM SINK actor for table slices.
  ::extend_with<stream_sink_actor<table_slice>>
  // Conform to the protocol of the STATUS CLIENT actor.
  ::extend_with<status_client_actor>;

That is, an analyzer must use CAF streaming to implement its functionality.

Implementing the Plugin

After having the necessary CMake in place, you can now derive from one or more plugin base classes to define your own plugin. Based on the chosen plugin types, you must override one or more virtual functions with an implementation of your own.

The basic anatomy of a plugin class looks as follows:

class example_plugin final : public virtual analyzer_plugin,
                             public virtual command_plugin {
public:
  /// Loading logic.
  example_plugin();

  /// Teardown logic.
  ~example_plugin() override;

  /// Initializes a plugin with its respective entries from the YAML config
  /// file, i.e., `plugin.<NAME>`.
  /// @param config The relevant subsection of the configuration.
  caf::error initialize(data config) override;

  /// Returns the unique name of the plugin.
  const char* name() const override;

  // TODO: override pure virtual functions from the base classes.
  // ...

private:
  record config_;
};

The plugin constructor should only perform minimal actions to instantiate a well-defined plugin instance. In particular, it should not throw or perform any operations that may potentially fail. For the actual plugin ramp up, please use the initialize function that processes the user configuration. The purpose of the destructor is to free any used resources owned by the plugin

Each plugin must have a unique name. This returned string should consicely identify the plugin internally.

Please consult the documentation specific to each plugin type above to figure out what virtual function need overriding. In the above example, we have a command_plugin and a analyzer_plugin. This requires implementing the following two interfaces:

system::analyzer_plugin_actor
make_analyzer(system::node_actor::pointer node) const override;

std::pair<std::unique_ptr<command>, command::factory>
make_command() const override;

After completing the implementation, you must now register the plugin with a specific version. VAST records the plugin version as 4-tuple with a major, minor, patch, and tweak version:

extern "C" struct plugin_version {
   uint16_t major;
   uint16_t minor;
   uint16_t patch;
   uint16_t tweak;
 };

We provide a macro to register the plugin with a specific version. For example, to register the example plugin with version 1.0.0-0, include the following line after the plugin class definition:

// This line must not be in a namespace.
VAST_REGISTER_PLUGIN(vast::plugins::example_plugin, 0, 1, 0, 0)

Registering Type IDs

The example plugin also shows how to register additional type IDs with the actor system configuration, which is a requirement for sending custom types from the plugin between actors. For more information, please refer to the CAF documentation page Configuring Actor Applications: Adding Custom Message Types.

Processing Configuration Options

To configure a plugin at runtime, VAST first looks whether the YAML configuration contains a key with the plugin name under the top-level key plugins. Consider our example plugin with the name example:

plugins:
  example:
    option: 42

Here, the plugin receives the record {option: 42} at load time. A plugin can process the configuration snippet by overriding the following function of vast::plugin:

caf::error initialize(data config) override;

VAST expects the plugin to be fully operational after calling initialize. Subsequent calls the implemented customization points must have a well-defined behavior.

Building the Code

Building a plugin requires a source tree of VAST. When configuring the build, you need to tell CMake the path to the plugin source directory. The CMake variable VAST_PLUGINS holds a semicolon-separated list of absolute paths to plugin directories. When using the provided configure script, use the option --with-plugin=path/to/plugin. Use this option multiple times when compiling more than one plugin.

After the compilation succeeded, you can find the shared object in lib/vast/plugins within your build directory.

To test that VAST loads the plugin properly, you need to enable it via the configuration first. We recommend to write a small test.yaml in your build directory with the following contents:

vast:
  plugin-dirs:
    - lib/vast/plugins
  plugins:
    - libexample

Then use vast --config=test.yaml version and look into the plugins. A key-value pair with your plugin name and version should exist in the output.

Unit and Integration Testing

VAST comes with unit and integration tests. So does a robust plugin implementation. We now look at how you can hook into the testing frameworks.

Unit Tests

Every plugin ideally comes with unit tests. We recommend creating a separate binary that links against both the plugin and libvast::test. The example plugin ships with dummy unit tests and the necessary CMake scaffolding.

Our convention is to name the test binary *-tests with * being the plugin name. You can find the test binary in bin within your build directory.

To execute registered unit tests, you can also simply run the build target test.

Integration Tests

Every plugin ideally comes with integration tests as well. Our convention is that integration tests reside in an integration subdirectory. If you add a file called integration/tests.yaml, VAST runs them alongside the regular integration tests. Please refer to the example plugin directory for more details.

Running the additional plugin integration tests requires a Pyton virtual env:

# Setup python3 venv (only needed once)
python3 -m venv env
source '<path/to/build>/vast/integration_env/bin/activate'
pip install -r 'vast/integration/requirements.txt'
pip install pyarrow # skip if built `--without-arrow`

# Run plugin-specific integration tests
source '<path/to/build>/vast/plugin_integration.sh'

Note that plugins may affect the overall behavior of VAST. Therefore we recommend to to run all integrations regularly by running the build target integration.

Packaging

If you plan to publish your plugin, you may want to create a GitHub repository. Please let us know if you do so, we can then link to community plugins from the documentation.

If you think the plugin provides a core functionality that is beneficial to all VAST users, feel free to submit a pull request to the main VAST repository.

Experimental Feature

Implementation Status

Usage#

Installating a Plugin#

Listing All Plugins#

Design#

Developing a Plugin#

The Scaffolding#

Choosing a Plugin Type#

Dreaded Diamond

Command Plugin#

Analyzer Plugin#

Implementing the Plugin#

Registering Type IDs

Processing Configuration Options#

Building the Code#

Unit and Integration Testing#

Unit Tests#

Integration Tests#

Packaging#

Usage

Installating a Plugin

Listing All Plugins

Design

Developing a Plugin

The Scaffolding

Choosing a Plugin Type

Command Plugin

Analyzer Plugin

Implementing the Plugin

Processing Configuration Options

Building the Code

Unit and Integration Testing

Unit Tests

Integration Tests

Packaging