mu server: add ID to requests to make them stateless

[Ambrevar]

I'm trying to add support to live updates of conversation buffers to mu4e-conversation and I'm facing some issues with regard to communicating with mu server.

As I understand it, each request sent to the server will result in one or more s-exps.
mu4e~proc-send-command sends the request, Emacs resumes its activity until it idles and thenmu4e~proc-filter catches the s-exps when they are emitted.

My current strategy to send requests tailored to mu4e-conversation is to advise the filter handlers (e.g. mu4e~headers-found-handler) with a custom function (e.g. collect all the thread messages in a structure for future display). I add the advice when sending the commands and I remove it from the advice itself.

While this works in the simple case, it remains brittle in the more general case as it leads to race conditions: when sending requests A then B, expecting answers A' and B' respectively, if A' does not get parsed before B is sent, the advice will be (wrongfully) removed before B' can be parsed.

(This is akin to client-server networking problems.)

One solution around would be to make everything stateless by adding request IDs to the requests, so that each command has a unique mapping to a set of requests.
This way it would be easy to specialize handlers for specific commands only.

It would be even nicer to make everything functional, i.e. get the s-exps as a return value of the command. I'm not sure if it's possible to do that however.

Insights?

[Ambrevar]

Looking at the broader picture: why do we need a long running process anyways (mu server)? It would be much simpler to call one process per query, then we could get the output directly from the call, i.e. everything would be functional.

Is this a performance concern?
time echo "cmd:view docid:47304�" | mu server finishes in <100ms, this should be good enough I think. What do you think?

[Ambrevar]

Forget my last comment, it can't work: the headers update take almost a second and Emacs hangs for that long every mu4e-update-interval. It's not acceptable, so we need an asynchronous process.

[djcb]

Keeping around a running mu-server is for two reasons:

1. speed -- think slightly older computer / non-ssd / huge Xapian database
2. locking the database, so we have less worries about mu4e displaying outdated information
   Of course things can always be reconsidered, but since the it generally works fine for mu4e, I'm not in hurry to fundamentally change this :-)
3. and yeah, as you say, some operations can take quite some time

As for IDs... dunno, it shouldn't be too hard (but adds some more complexity to the server); however, the server is synchronous, so you get back your answers (possibly multiple per command) in exactly the same order as you send commands.

Generally it's a bit tricky to depend on various mu4e implementation details, which could change at any time; and especially changing those details for some external use-case. E.g., I can foresee that once we can overcome the problems with the gnus-based viewer, that could replace the existing viewer. With that in place; and then we could get rid of cmd:view completely (since the gnus-based method only needs a :path, which we already have in the headers-info for a message).

[Ambrevar]

Acknowledged.

IDs are always a good idea as they allow the client to map between requests and responses.
When working on extending mu4e, the need arises very quickly.

I've implemented a (somewhat hacky) implementation of it with https://gitlab.com/ambrevar/mu4e-conversation: look at mu4e-conversation--enqueue, mu4e-conversation--proc-filter and mu4e-conversation--query-thread.
It works very well so far. It allows for multiple requests to be run simultaneously (displaying / updating multiple thread in unpredictable order).

While it works it would be even better if the server would support IDs / callbacks out of the box.

[djcb]

Yeah, guess I agree with my 2018 answer; and to add to that, the comms protocol between mu and mu4e is not really the right place for third-party extensions; it does change fairly frequently (I just did a bunch of changes), for optimization, new features etc., and I really don't want any backward-compat headaches there.

There are some newer tickets about third-party APIs, such as #2520 and #2210, so I guess the discussion can continue there.