Best pratices for large data sharing

In my VST3 SAM-SPL64 plugin, you “load” a sample (which can be of any size, including several Megabytes) which you can then “play” after being split in multiple slices.

I have implemented the behavior following what I believe are the guidelines for proper VST handling:

  • the GUI (and I am using VSTGUI throughout) is the one that loads the sample from the filesystem since it is accessing the filesystem (and that should not happen in the RT code)
  • the GUI then sends a message to the RT in the form of 2 buffers (if stereo) with the samples to play (using the EditController::sendMessage / ComponentBase::notify infrastructure)
  • the GUI also saves the buffer in the GUI state (EditController::setState) in order to be self contained (not relying on a local file that may disappear in the future).

At pretty much all point in time, the GUI and the RT both need the 2 buffers:

  • the GUI needs it because it displays it (as a waveform) and because it saves it in the GUI state
  • the RT needs it because it plays it when the user hits a pad in the UI or a midi key

This obviously causes a lot of churn and duplicated memory:

  • the GUI and the RT have a different copy
  • when the GUI sends the buffer to the RT, it gets “serialized” and “deserialized” also copying large amounts of data

Obviously it works and like I said seems to be the “best” practice / recommended way… But in practice this seems incredibly wasteful. In “theory” the plugin can be distributed (flag ComponentFlags::kDistributable), but in practice, how many DAW actually use this feature?

I know and understand that in theory this is a very good model which clearly separates the GUI and RT especially when it comes to sharing data without locking which can be incredibly hard to get right, but again, in this very specific case, “purity” comes at a hefty price.

So my questions are:

  • did I read/interpret the guidelines correctly in terms of how to handle data?
  • would you recommend a different way of doing it?
  • do you have a sense of how many DAWs actually physically distribute the GUI and RT of a plugin?
  • since the flag ComponentFlags::kDistributable is clearly just an option at the moment, do you foresee a moment in a future SDK when it would be mandated and no longer an option?
  • I am seriously thinking about a way to share the buffer between the UI and RT, am I crazy?

Thanks
Yan

Hi Yan,
you are doing it correctly.
You should always work on separate data sets in your controller and your processor. Not only because they may be in different processes, but also because of delay compensation, when your processor is called in advance to your controller and you want to sync the audio to the visual representation.

Cheers,
Arne