how to use own GUI / windows instead VSTI GUI?

The way I have done this is to subclass CPluginView and then override the base class functions that you need:

IPlugFrame* plugFrame = nullptr;
const ViewRect& getRect() const { return rect; }
void setRect(const ViewRect& r) { rect = r; }
bool isAttached() const { return systemWindow != 0; }

virtual void attachedToParent() override {}
virtual void removedFromParent() override {}
virtual tresult PLUGIN_API attached(void* parent, FIDString type) override;
virtual tresult PLUGIN_API removed() override;
virtual tresult PLUGIN_API onWheel(float distance) override { return kResultFalse; }
virtual tresult PLUGIN_API isPlatformTypeSupported(FIDString type) override;
virtual tresult PLUGIN_API onSize(ViewRect* newSize) override;
virtual tresult PLUGIN_API getSize(ViewRect* size) override;
virtual tresult PLUGIN_API onFocus(TBool /state/) override { return kResultFalse; }
virtual tresult PLUGIN_API setFrame(IPlugFrame* frame) override;// { plugFrame = frame; return kResultTrue; }
virtual tresult PLUGIN_API canResize() override{ return kResultTrue; }
virtual tresult PLUGIN_API checkSizeConstraint(ViewRect* rect) override;
virtual bool setWindowFrameSize(double left = 0, double top = 0, double right = 0, double bottom = 0) override //CRect* newSize)
virtual bool getWindowFrameSize(double& left, double& top, double& right, double& bottom) override

As for the “two DLL” issue, I am guessing you are wanting to link to a 3rd party DLL at runtime?

I’ve linked VST3s with both static and dynamic libraries many times, and you need to watch out for the common problems – in general, there may be conflicting libraries (“Ignore Specific Default Libraries” in VS) and be careful with namespacing. I’ve you’ve linked DLLs to other API’s then you’ve been through those issues already.

I’ve mixed VST2 and VST3 with MFC (for windows and views) in the past – it is not very much fun, but it is completely do-able.

That said, I’ve been using VSTGUI4 since 2011 and have fully switched over now – it is sweet to be able to use the same code for AU, VST, and AAX on MacOS and Windows, but my GUI is also “custom” in the sense that it subclasses CPluginView in order to implement the GUI operation.

Hope that helps.

Will