HTML/JS user interfaces

[lucianoiam]

Hi @falkTX. I've been working on this for a while https://github.com/lucianoiam/dpf-webui and wondering if it makes sense to integrate it into DPF as one of the UI_TYPEs . While it is overkill for typical plugin work consisting of knobs and switches, It could be interesting for building more app-like complex UIs (thinking of the various JS frameworks out there). I personally dislike webapps but in contrast to let's say Electron, this is not about embracing JS for everything but rather adding some bits of its goodness to the already wild world of plugin UIs. And of course leveraging a full web rendering engine for flexbox, canvas, SVG, etc. It could work for some specific cases while following the DSP/UI separation paradigm DPF enforces.

A more generic, perhaps better integration could involve turning the web view into one of DPF's native Widgets. Then explain the web-based user interface approach in one of the example projects. This breaks dpf-webui into two separate tools: a reusable webview and a JS bridge to DISTRHO::UI.

I think of dpf-webui as a naive but working implementation of a (debatable) good idea which is throwing some higher level language pieces on top of the audio engine core like Bitwig does, or a quite recent project called React-JUCE. While drawing from DPF's spirit of not complicating things.

The interfaces to the system web views were written from scratch to ensure they are safe to run in a plugin context on all platforms. No dependencies needed.

Being said this, thanks to the DPF team for all the great work.

[falkTX]

Hello and thanks for all the work!

I am a bit apprehensive about HTML on plugin UIs, but maybe indeed there is sometimes a good use for them.
The way to go on here that I really recommend, is to try with DPF develop branch and make the WebView stuff be a subclass of TopLevelWidget (this is a new thing in develop branch).
I basically took the things that made the "UI" class special, and put them as a TopLevelWidget. So it can be used not just for plugins but also standalones.

Quick look through the code, I have a few more comments:

1. use typedef's instead of macros, for example for DISTRHO_WEBVIEW_CLASS
2. there is a native API to find scale factor in DPF now. it is only implemented for plugins right now, standalones can use an env var, but best to switch to that
3. switch the webview to inherit from TopLevelWidget instead of UI class
4. it is possible to dlopen the standalone executable, see https://stackoverflow.com/questions/19713517/how-to-self-dlopen-an-executable-binary

[lucianoiam]

Thanks for having a look and for the tips! I see some hot progress going on develop and no. 3 sounds like the right way to support the web view.

Off-topic I see a lot of value in the standalone target, I recall reading it was originally created for rapidly testing plugins. Please give it the same treatment as the plugin targets (I think there have been already improvements in that department lately). The only "bottleneck" is JACK which is a bit alien to anything non Linux. Bottleneck because there is so much heavy underlying work for Mac/Win that can get potentially shadowed by the fact, and the double quotes because I love JACK and use it on a daily basis -- but that is not true for my musician friends and probably the average audio-oriented user. Is it a decent idea to bundle a "private, invisible to foreign processes" JACK server into the standalone version? likely the server needs modifications. Or is there any portable audio library that outputs to JACK as an option apart from the system API? the latter sounds interesting as you'd get the best from all worlds. I feel I'm not inventing the wheel but cannot recall seeing that.

[falkTX]

I changed DPF recently so that JACK libs/headers are not needed to build the JACK target, it will try to dlopen libjack at the start.
At some point we could get a native dialog popup indicating JACK is not available or started.

The problem with doing something outside of JACK for standalones is that it would need user options to setup device, buffer size, sample rate, midi devices etc..
This goes against the idea of DPF being sorta minimalistic, and I do not plan to add such things at all.
(A separate project will make this possible though, loading DPF plugins and treating the whole deal as standalone https://github.com/falkTX/Chibi)

JACK for audio plugin standalones is fine IMO, I am maintaining it so that it still works on macOS and Windows, so it is not hard to get things working IMO, just needs to be installed and running.

When I speak of standalones I am not just referring to audio plugins though, but using DPF DGL things as actual GUI library for an application (without audio).
Having HTML support means electron-like applications become possible, but without the heavy electron engine behind it.

[falkTX]

btw, develop DPF has a "stub" target now too. something that does not setup openGL, cairo or any graphics backend.
Once you have the webview as a toplevelwidget, I will be interested to see if we could make it work with the "stub" backend, so we can skip unnecessary opengl things.

[lucianoiam]

DPF minimalism if one of its greatest strengths and agree on that. The plugin shell approach like Chibi sounds a lot better, I was not aware of it. The best of these things and it also applies to JACK is being modular and they should remain the same.

I agree on JACK not hard to get working but er... you know people sometimes are lazy :). I have some ideas to help on that but it would be dishonest to continue the rant because I am lazy myself to implement them. Just bringing up the point that an extra step in the workflow bothers some poeple and that is out of the developers control, especially on those systems where everything is "automatic". No wonder why Pro Tools looks basically the same as 30 years ago.

In fact dpf-webui was conceived as the starting point for a port of a custom modular system developed in the past based on Python, the hacked Live Live API, HTML/JS, some audio utilities and JACK for inter-app audio. Replicating the setup on someone else's computer proved to be non trivial. It occurred to me that if I tried to (forgive me) glue everything together into a VST plugin it would make it easier to share the thing with bandmates and eventually release it to the general public. So the workflow remains the same for them, start the DAW and forget. Audio routing is done inside the DAW, the visual interface can be casted to the network, etc. It is a very specific case.

I saw the "stub" target yesterday and had a vague idea, the comment confirms I definitely need to dig into it. The web view only needs a pointer to the native parent view, thanks !

[falkTX]

well worst case for audio plugin standalones would be a different target, I think SDL would work well.
so we could try JACK if available, and if not fallback to SDL but still keep the jack API intact.

[lucianoiam]

Now I think there is no need for a sound configuration panel, if it worked like that then for the user it would be automagical. If they want to get benefited from studio-like facilities the more advanced users will install JACK, while keeping it simple for others (install - run - profit).

[lucianoiam]

There is a new branch https://github.com/lucianoiam/dpf-webui/tree/moredpf . The class hierarchy looks like this now:

TopLevelWidget ← BaseWebWidget ← PlatformWebWidget ( the latter now a typedef, thanks )

I naively tried switching UI_TYPE to stub but the project won't compile so there is clearly something I am overlooking. I am still not familiar with the DPF design; everything built so far was done on top of it and not hooking into it.

The top level UI for this project was implemented as a derived class of DISTRHO::UI ( called DISTRHO::WebUI ) because its goal is acting as a user interface to the Plugin, ie. call setParameterValue(), listen to parameterChanged(), and so on. DISTRHO::WebUI is a controller for the web widget that basically does two things 1) display the web widget 2) act as a proxy to JS class DISTRHO_WebUI, leveraging a minimalistic generic bridge provided by the web widget. The web widget itself is "dumb".

Following your convention, isn't DISTRHO::WebUI the top level widget of this design? why *WebWidget should derive from TopLevelWidget and not Widget? . Is it something related to the graphical backend?

Summarizing in inaccurate terms, this is what I'm thinking of:

1. Have a DISTRHO::UI that skips the graphical backend but still provides the raw native window. This could be useful for other blasphemic variations of DPF. Is this what 'stub' is for ?
2. Ideally make the WebWidgets suitable for adding as subwidgets anywhere so they are not limited to what dpf-webui does. There's some potential there even beyond rendering, like scripting or network (the latter needs some adjustments).

In file included from ./lib/DPF/distrho/src/DistrhoUI.cpp:17:
./lib/DPF/distrho/src/DistrhoUIPrivateData.hpp:232:29: error: base class has incomplete type
class PluginWindow : public Window
                     ~~~~~~~^~~~~~
./lib/DPF/dgl/Widget.hpp:30:7: note: forward declaration of 'DGL::Window'
class Window;
      ^
In file included from lib/DPF/distrho/DistrhoUIMain.cpp:17:
In file included from ./lib/DPF/distrho/src/DistrhoUI.cpp:17:
./lib/DPF/distrho/src/DistrhoUIPrivateData.hpp:242:76: error: only virtual member functions can be marked 'override'
    void onFocus(const bool focus, const DGL_NAMESPACE::CrossingMode mode) override
                                                                           ^~~~~~~~
...

[falkTX]

1. In turn there is a single OpenGL (or whatever) view that supports the entire DGL "canvas" so any foreign stuff added on top will obscure DGL, right?

That is correct. The webview being on top of the other widgets, even if rendered before or after, is not something we can change because this stack is from the OS-level.
The only way around it would be to create other OS-level embed windows/widgets/views and put those on top of the webview.
This is the kinda stuff I want to avoid adding into DPF, the complexities on this rise up pretty quickly..

I can understand wanting to position the web widget on a specific area of the window though.
Maybe we can have the webview still as top-level-widget, but have a special flag to tell the window to not resize it automatically for us. (though this behaviour should be optional, as it is useful to auto-resize to full width/height if for example UI type is html)

It just doesnt make sense to have this as a SubWidget if it is never really going to behave like one.

[lucianoiam]

Well actually for my own specific use case ( a plugin with a full blown HTML UI ), there is no need for web views < window area; this was more like an exercise and helped learning from DPF and refining the widgets. Repositioning is not probably very useful. However I think I am still missing something and that SubWidget vs TopLevelWidget is not only about dimensions. As far as I understand, many of the issues described also hold true if the widget was derived from TopLevelWidget, like not being able to contain subwidgets. What else would be needed to behave as a SubWidget? just curious, I don't think I'll update the widgets much more now.

Agree on the complexity of embedding and IMO the use cases are a few even for general purpose frameworks.

[falkTX]

As far as I understand, many of the issues described also hold true if the widget was derived from TopLevelWidget, like not being able to contain subwidgets

We can try to find workarounds for that, but yes that is still an issue.

What else would be needed to behave as a SubWidget?

Integrating in the current running backend (opengl or cairo) most likely.
So if we need to do something like a scrolling page of widgets, to be able to.

In reality the webview is a foreign widget.

tbqh I would prefer to have the webview be only allowed if ui type is html, with the stub stuff, and not mix in any cairo or opengl.
but for that we still need that stub thing to work properly.

it all depends on what you want to do with it of course. if an opengl plugin wants to show some documentation built-in, I guess it makes sense.
but there is always caveats...

[lucianoiam]

As far as I understand, many of the issues described also hold true if the widget was derived from TopLevelWidget, like not being able to contain subwidgets

We can try to find workarounds for that, but yes that is still an issue.

IMO that could be just unsupported, it is very complex and unnecessary. The challenge is how to make it elegantly unsupported.

What else would be needed to behave as a SubWidget?

Integrating in the current running backend (opengl or cairo) most likely.
So if we need to do something like a scrolling page of widgets, to be able to.

True, the current SubWidget-based implementation would not behave properly in a scroll view.

In reality the webview is a foreign widget.

tbqh I would prefer to have the webview be only allowed if ui type is html, with the stub stuff, and not mix in any cairo or opengl.
but for that we still need that stub thing to work properly.

it all depends on what you want to do with it of course. if an opengl plugin wants to show some documentation built-in, I guess it makes sense.
but there is always caveats...

Yes undoubtedly this will never play completely well in the DGL arena. I would leave that up to you, how to eventually include the widget in DPF if finally decided to. For my own use case I'd say it already does more than I initially expected. Also there is a non-trivial aspect of the integration which is how to build the views. Each platform has some details needing attention, everything is in the dpf-webui Makefiles which are of acceptable quality but fitting it into the framework's own build scripts requires some effort.

[lucianoiam]

Did you try to force gtk x11 backend?
Since the gtk stuff is in a separate process, this wont conflict with any running tools.
I know you can at least set an environment variable to set gtk backends, maybe possible with code too.

That's right. I did some research, it is also possible with code:

gdk_set_allowed_backends("x11,wayland");

The order defines which backend is tried first.

Now the GTK-based helper app thinks there's X11 but XReparentWindow() does not seem to work, the window to be embedded just disappears after the call. Nevermind, I don't have the time to deal with that. Thanks for the tip!

[falkTX]

So to be clear, it is best that webview is a top-level-widget, so we do not have to deal with child windows, partial embedding and stuff like that.
When using web stuff to render a widget, IMO it should be purely web/html and not mix cairo and opengl stuff with it.

Basically, think of Web/HTML as a target backend within DPF - So we get Cairo, OpenGL or Web/HTML.

[lucianoiam]

I think the difference between the two approaches is the scope of the web widget. Should it be a reusable component, or exclusively limited to a way of supporting completely web-based user interfaces? the dpf-webui project started with the latter goal in mind but then after some discussion, it took little effort to turn the web widget (90% of the project effort) into a reusable component.

[lucianoiam]

It's been some time since this, finally the project evolved into web UI support (JavaScript) and optionally web DSP as well (AssemblyScript). More or less at beta stage at this point repo here. The result is pretty solid, there is a single thing I could still not fix but related to WebKitGTK and not DPF. There is a workaround in place but it breaks CSS viewport-based units on Linux.

The current webview version finally subclasses SubWidget for historic (read: lazy) reasons but now #313 looks like the best approach.

Also just discovered https://github.com/LDPF , referencing relevant discussion here LDPF@4cbed01

[falkTX]

Great stuff!
And yeah, the embed-external ui might be the best way around this. Before I thought that we might need the stub stuff, but on 2nd thought that is quite pointless since for example webgtk already creates its own X11 window.

[lucianoiam]

That's correct. There is a single caveat, maybe you know something about this. The GTK webview must be attached to a GtkWindow. Since the x11 window can already exist (ie, provided by the host) it is wrapped like this: GdkWindow* gdkChild = gdk_x11_window_foreign_new_for_display(gdk_display_get_default(), x11_window);

This works great but the webview does not seems to follow this special Gtk/GdkWindow resize events afterwards. Not to blame DPF, maybe you have experience wrapping x11 windows for use in Gtk and there are known limitations when "adopting" windows. In the meantime I filed a bug at the WebKitGTK tracker.

The current workaround is to create the webview with fixed large dimensions (like 2000x2000), it will be cropped by the wrapping window. And then use CSS for manipulating the body dimensions. As ugly as it sounds it works very good but it prevents vh/vw/etc from working correctly. This only applies to Linux.

[falkTX]

Dont you need a container to setup the sizes right?
https://github.com/lv2/suil/blob/master/src/x11_in_gtk2.c might have some useful code, not sure.
I dont typically deal with gtk.

[lucianoiam]

Thanks for the snippet will have a look at it. Resizing works as expected when creating a "normal" standalone GtkWindow and attaching the webview to it with no extra effort, it just works. There has to be a difference between a regular GtkWindow and the wrapped x11 version, already discovered that the latter does not emit some events maybe it is that. Not sure it is a supported setup though.

[lucianoiam]

Continuation of #316

I like the idea of integrating all webview development into DPF. I started the project out of necessity as the first step of a project that needs a complex UI and runs on all desktop platforms. As a side effect, later discovered some overlap with the growing territory of web audio projects.

I see two main approaches:

• Only distribute the source files and hand over build process to the plugin dev. Like if they were reusable pseudo-widgets. Switching to external means that they are not real widgets anymore, in the DGL sense.
• Treat web as a new UI choice that depends on external. Have DPF take care of the build process like the current separate webview project does. This requires merging Makefiles and replicating things on CMake, I have no experience with the latter.

Right after both parts (external-ui and webview) are complete sounds like the right time to study how to merge. The current webview can already produce working plugins that depend on DGL example here , and there is a separate no-DGL branch that is already complete, minus standalone.

I would start by completing external-ui, then probably work a little more on the UI class as explained below to pave the way for integration.

Sharing some thoughts including long term ideas. The webview project provides two main classes, AbstractWebHostUI and AbstractWebView:

• AbstractWebHostUI: manages the web view, C++/JS bridging of some UI methods, and also provides a minimal base for abstracting plugin vs standalone cases. Regarding the latter functionality, wouldn't it make sense to have that as a stock DPF feature? now we have external UI, I am quite sure other external UI devs will seek to abstract the plugin environment. For example, getParentWindowHandle() could return the host window or a minimal standalone window. I know this means repeating some DGL/Pugl code, but also potentially gives a more control to DPF. It would be something like UI_TYPE=native , or window. Long term: in the future DGL widgets could run on top of this stack, instead of the current inverse situation.

• AbstractWebView: these were previously Widgets but does not make sense for external so the new nodgl branch renamed them to *WebView and stripped off any dependency on DGL. Ideally we would keep these generic, reusable even outside DPF. Long term: make them really generic, multiplatform webviews are very sought after and can benefit from a broader community contributions. I think the only dependency on DPF right know is the String class

Then we have the WasmPluginHost but that is easier to handle because it does not require graphics. It is quite complete, including a subset of the DPF Plugin API ported to AssemblyScript. Long term: the only thing I am not sure about is keeping Wasmer as the Wasm engine, or switch to something minimal like WAMR. Time will tell, some stuff is even still discussed by the Wasm people.

Well there are some details to think about, tried to keep it short.