Plugins

Experimental Feature

This is an experimental feature: the API is subject to change and robustness is not yet comparable to production-grade features.

VAST has a plugin mechanism that makes it easy to hook into various places of the data processing pipeline and add custom extensions in a safe and sustainable fashion. A set of customization points allow anyone to add new CLI commands, receive a copy of the input stream, spawn queries, or implement integrations with third-party libraries.

Implementation Status

The light-yellow, dashed plugins in the above figure have not yet been implemented. Specifically, the missing plugin types are:

  1. Carrier: a data transport mechanism, like UDP or TCP
  2. Reader: an parser to transform a specific format into VAST data
  3. Writer: a printer to render VAST data in a specific format
  4. Import: a filter applied after having parsed the input at a source
  5. Export: a filter applied before passing query results to a sink

Usage

Installating a Plugin

There exist two ways to deploy plugins:

  1. Static: compiled into VAST. Native plugins part of the VAST code base ship in this form.
  2. Dynamic: loaded as separate shared library, typically used for third-party or binary-only plugins.

Static plugins do not require loading since they are compiled into VAST. Dynamic plugins are shared libraries and therefore must be loaded first into the running VAST process. At startup, VAST looks into the configured vast.plugin-dirs directories in the vast.yaml file to find the configured vast.plugins files.

vast:
plugin-dirs:
- .
- /opt/foo/lib
plugins:
- libexample
- /opt/foo/lib/libexample.so

Before executing functionality, VAST loads the specified plugins via dlopen(3) and attempts to initialize them as plugins. If the version check passes, VAST initializes the plugin with the respective configuration in the vast.yaml file:

plugins:
example:
option: 42

After initialization with the configuration options, the plugin is fully operational and VAST will call its functions at the plugin-specific customization points.

Listing All Plugins

You can get a list of all plugins and their respective version by running vast version:

{
"VAST": "2020.12.16-97-gced115d91-dirty",
"CAF": "0.17.6",
"Apache Arrow": "2.0.0",
"PCAP": "libpcap version 1.9.1",
"jemalloc": null,
"plugins": {
"example": "0.0.0-0"
}
}

Design

VAST's plugin design follows a few principles:

  • Complementary: plugins must be additive, no mutation of existing functionality.
  • Composable: a single plugin can offer multiple features, e.g., add a custom CLI command while simultaneously hooking into the import stream.
  • Configurable: users should be able to configure the different tuning knobs of a plugin in the vast.yaml.

We found that a class-based plugin hierarchy with name-based registration achieves these design goals, as long as the customization points act orthogonal to each other.

Developing a Plugin

Implementing a new plugin requires the following steps:

  1. Setup the scaffolding
  2. Choose a plugin type
  3. Implement the plugin interface
  4. Process configuration options
  5. Package it

Next, we'll discuss each step in more detail.

The Scaffolding

The scaffolding of a plugin includes the CMake glue that makes it possible to use as static or dynamic plugin.

VAST ships with an example plugin that showcases how a typical scaffold looks like. Have a look at the the example plugins directory.

Choosing a Plugin Type

VAST offers a variety of customization points, each of which defines its own API by inheriting from the plugin base class vast::plugin. When writing a new plugin, you can choose a subset of available types by inheriting from the respective plugin classes.

Dreaded Diamond

To avoid common issues with multiple inheritance, all intermediate plugin classes that inherit from vast::plugin use virtual inheritance to avoid issues with the dreaded diamond. When composing multiple plugin types, however, you should use non-virtual public inheritance only.

Please consult the example plugin for a concrete code sample.

Command Plugin

A command plugin adds a new command to the vast executable, at a configurable location in the command hierarchy.

The base class vast::command_plugin defines a factory function make_command() that returns a new command. Concretely, the return value of this function is a std::pair<std::unique_ptr<command>, command::factory>. The first component is the command instance, and the second defines the mapping from command name to command implementation.

Analyzer Plugin

The analyzer plugin hooks into the processing path of data by spawning a new actor inside the server that receives the full stream of table slices.

The base class vast::analyzer_plugin defines a typed actor interface analyzer_actor that users must return by overriding the pure virtual factory make_analyzer(caf::actor_system&). The analyzer_actor has the following type:

using analyzer_actor = caf::typed_actor<
caf::reacts_to<caf::stream<table_slice>>
>;

That is, an analyzer must use CAF streaming to implement its functionality.

Implementing the Plugin

After having the necessary CMake in place, you can now derive from one or more plugin base classes to define your own plugin. Based on the chosen plugin types, you must override one or more virtual functions with an implementation of your own.

The basic anatomy of a plugin class looks as follows:

class example final : public virtual analyzer_plugin,
public virtual command_plugin {
public:
/// Loading logic.
example();
/// Teardown logic.
~example() override;
/// Initializes a plugin with its respective entries from the YAML config
/// file, i.e., `plugin.<NAME>`.
/// @param config The relevant subsection of the configuration.
caf::error initialize(data config) override;
/// Returns the unique name of the plugin.
const char* name() const override;
// TODO: override pure virtual functions from the base classes.
// ...
private:
record config_;
};

The plugin constructor should only perform minimal actions to instantiate a well-defined plugin instance. In particular, it should not throw or perform any operations that may potentially fail. For the actual plugin ramp up, please use the initialize function that processes the user configuration. The purpose of the destructor is to free any used resources owned by the plugin

Each plugin must have a unique name. This returned string should consicely identify the plugin internally.

Please consult the documentation specific to each plugin type above to figure out what virtual function need overriding. In the above example, we have a command_plugin and a analyzer_plugin. This requires implementing the following two interfaces:

analyzer_actor make_analyzer(caf::actor_system& sys) const override;
std::pair<std::unique_ptr<command>, command::factory>
make_command() const override;

After completing the implementation, you must now register the plugin with a specific version. VAST records the plugin version as 4-tuple with a major, minor, patch, and tweak version:

extern "C" struct plugin_version {
uint16_t major;
uint16_t minor;
uint16_t patch;
uint16_t tweak;
};

We provide macro to register the plugin with a specific version. For example, to register the example plugin with version 1.0.0-0, include the following line after your plugin class definition:

VAST_REGISTER_PLUGIN(example, 0, 1, 0, 0)

Processing Configuration Options

To configure a plugin at runtime, VAST first looks whether the YAML configuration contains a key with the plugin name under the top-level key plugins. Consider our example plugin with the name example:

plugins:
example:
option: 42

Here, the plugin receives the record {option: 42} at load time. A plugin can process the configuration snippet by overriding the following function of vast::plugin:

caf::error initialize(data config) override;

VAST expects the plugin to be fully operational after calling initialize. Subsequent calls the implemented customization points must have a well-defined behavior.

Packaging

If you plan to publish your plugin, you may want to create a GitHub repository. Please let us know if you do so, we can then link to community plugins from the documentation.

If you think the plugin provides a core functionality that is beneficial to all VAST users, feel free to submit a pull request to the main VAST repository.