VAST offers a flexible way of distributing its key components over processes, thanks to the actor model. This flexibility accommodates a wide range of possible deployment scenarios, ranging from highly integrated single-process applications to distributed setups.
When ingesting data, there are two options to spin up sources: either run them
as a separate client process via
vast import or spawn them directly inside
the server process via
vast spawn. We discuss both options below.
Source as Client Process
By default, starting VAST only spawns the necesssary server components, such as archive and index. This works well for ad-hoc scenarios where users perform manual imports or for low-volume continous data sources. The VAST client process connects to the server and sends the parsed data over a TCP connection. The setup looks like this:
Source within Server Process
The second method of ingesting data involves spawning a source actor directly inside the server process. The advantage is performance: it cuts out any IPC. Sending parsed data from the source is equivalent to passing a pointer in memory. For high-volume formats, like PCAP or NetFlow, avoiding stress on the I/O path can make a major difference.