Plugins
Experimental Feature
This is an experimental feature: the API is subject to change and robustness is not yet comparable to production-grade features.
VAST has a plugin mechanism that makes it easy to hook into various places of the data processing pipeline and add custom extensions in a safe and sustainable fashion. A set of customization points allow anyone to add new CLI commands, receive a copy of the input stream, spawn queries, or implement integrations with third-party libraries.
Implementation Status
The light-yellow, dashed plugins in the above figure have not yet been implemented. Specifically, the missing plugin types are:
- Carrier: a data transport mechanism, like UDP or TCP
- Reader: an parser to transform a specific format into VAST data
- Writer: a printer to render VAST data in a specific format
- Import: a filter applied after having parsed the input at a source
- Export: a filter applied before passing query results to a sink
Usage
Installating a Plugin
There exist two ways to deploy plugins:
- Static: compiled into VAST. Native plugins part of the VAST code base ship in this form.
- Dynamic: loaded as separate shared library, typically used for third-party or binary-only plugins.
Static plugins do not require loading since they are compiled into VAST.
Dynamic plugins are shared libraries and therefore must be loaded first into
the running VAST process. At startup, VAST looks into the configured
vast.plugin-dirs
directories in the vast.yaml
file to find the configured
vast.plugins
files.
Before executing functionality, VAST loads the specified plugins via dlopen(3)
and attempts to initialize them as plugins. If the version check passes, VAST
initializes the plugin with the respective configuration in the vast.yaml
file:
After initialization with the configuration options, the plugin is fully operational and VAST will call its functions at the plugin-specific customization points.
Listing All Plugins
You can get a list of all plugins and their respective version by running
vast version
:
Design
VAST's plugin design follows a few principles:
- Complementary: plugins must be additive, no mutation of existing functionality.
- Composable: a single plugin can offer multiple features, e.g., add a custom CLI command while simultaneously hooking into the import stream.
- Configurable: users should be able to configure the different tuning
knobs of a plugin in the
vast.yaml
.
We found that a class-based plugin hierarchy with name-based registration achieves these design goals, as long as the customization points act orthogonal to each other.
Developing a Plugin
Implementing a new plugin requires the following steps:
- Setup the scaffolding
- Choose a plugin type
- Implement the plugin interface
- Process configuration options
- Package it
Next, we'll discuss each step in more detail.
The Scaffolding
The scaffolding of a plugin includes the CMake glue that makes it possible to use as static or dynamic plugin.
VAST ships with an example plugin that showcases how a typical scaffold looks like. Have a look at the the example plugins directory.
Choosing a Plugin Type
VAST offers a variety of customization points, each of which defines its own
API by inheriting from the plugin base class vast::plugin
. When writing a new
plugin, you can choose a subset of available types by inheriting from
the respective plugin classes.
Dreaded Diamond
To avoid common issues with multiple inheritance, all intermediate plugin
classes that inherit from vast::plugin
use virtual inheritance to avoid
issues with the dreaded
diamond. When
composing multiple plugin types, however, you should use non-virtual public
inheritance only.
Please consult the example plugin for a concrete code sample.
Command Plugin
A command plugin adds a new command to the vast
executable, at a configurable
location in the command hierarchy.
The base class vast::command_plugin
defines a factory function
make_command()
that returns a new command. Concretely, the return value of
this function is a std::pair<std::unique_ptr<command>, command::factory>
. The
first component is the command instance, and the second defines the mapping
from command name to command implementation.
Analyzer Plugin
The analyzer plugin hooks into the processing path of data by spawning a new actor inside the server that receives the full stream of table slices.
The base class vast::analyzer_plugin
defines a typed actor interface
analyzer_actor
that users must return by overriding the pure virtual factory
make_analyzer(caf::actor_system&)
. The analyzer_actor
has the following
type:
That is, an analyzer must use CAF streaming to implement its functionality.
Implementing the Plugin
After having the necessary CMake in place, you can now derive from one or more plugin base classes to define your own plugin. Based on the chosen plugin types, you must override one or more virtual functions with an implementation of your own.
The basic anatomy of a plugin class looks as follows:
The plugin constructor should only perform minimal actions to instantiate a
well-defined plugin instance. In particular, it should not throw or perform any
operations that may potentially fail. For the actual plugin ramp up, please use
the initialize
function that processes the user configuration. The purpose of
the destructor is to free any used resources owned by the plugin
Each plugin must have a unique name. This returned string should consicely identify the plugin internally.
Please consult the documentation specific to each plugin type above to figure
out what virtual function need overriding. In the above example, we have a
command_plugin
and a analyzer_plugin
. This requires implementing the
following two interfaces:
After completing the implementation, you must now register the plugin with a specific version. VAST records the plugin version as 4-tuple with a major, minor, patch, and tweak version:
We provide macro to register the plugin with a specific version. For example,
to register the example
plugin with version 1.0.0-0
, include the following
line after your plugin class definition:
Processing Configuration Options
To configure a plugin at runtime, VAST first looks whether the YAML
configuration contains a key with the plugin name under the top-level key
plugins
. Consider our example plugin with the name example
:
Here, the plugin receives the record {option: 42}
at load time. A plugin can
process the configuration snippet by overriding the following function of
vast::plugin
:
VAST expects the plugin to be fully operational after calling initialize
.
Subsequent calls the implemented customization points must have a well-defined
behavior.
Packaging
If you plan to publish your plugin, you may want to create a GitHub repository. Please let us know if you do so, we can then link to community plugins from the documentation.
If you think the plugin provides a core functionality that is beneficial to all VAST users, feel free to submit a pull request to the main VAST repository.