Async exception tracking

[nitely]

Somewhat inspired in chronos.

Goal:

• Enable Exception Tracking for async. Using the native Nim exception tracking.
• Enable passing error typed Futures around.
• No runtime overhead.
• ~Backward compatible.

How:

• Await retrieves the raises list for the future and sets it as raises for the read call. If raises cannot be retrieved, then nothing is set, and the current behavior is kept (raises: [Exception]).
• toFutureEx macro takes an async function call, and it returns the error typed future FutureEx[T, E] for passing it around.
• addCallback callbacks must not raise for this to work. It's possible to make this backward compatible, I think. Just deprecate the proc that takes a cb without raises: [], and cast it to untrack raises.
• newFuture() usage in user code usually leads to inferred Exception at some point because read(Future) can raise anything. If the user wants granular raises, they must use FutureEx.

Todo:

• tests
• implement all, and, or FutureEx versions
• implement fail FutureEx version; can it be done so it only accepts the typed exceptions?
• fix all XXX
• check all nimble packages using async compile

Status: main blocker to move this forward is I cannot detect if a type is an async proc, or a closure that returns a Future (they are both the same type right now). See this test if we pass an async proc, the raises get inferred as [Exception] as well, since we cannot check if the callback is an async proc or a closure returning Future that can eventually fail with any exception. I suspect it may not be possible to get the raises lists for async generic callbacks.

[nitely]

I'm almost certain this cannot be done in a backward compatible way. A lot of (non-async) procs in asyncdispatch.nim create and return a newFuture[T], and so they raises: [Exception]. These procs would need to return a FutureEx[T, E] to make error tracking granular. One way forward is to create an asyncdispatch2.nim. Make all procs wrappers of the error typed versions, and deprecate them. But that's pretty bad, and a lot of work.

Maybe making FutureEx*[T, E] = ref object of Future[T] instead of FutureEx*[T, E] = distinct Future[T] would cause less breakage in practice. But maybe asyncdispatch2.nim would still be needed because we cannot know how these are used outside of the nimble packages.

I do realize this is throwing the baby out with the bathwater. The user could handle the asyncdispatch Exception raises, and have sane error tracking in the rest of their code. But I don't see users requesting async error tracking either.

[arnetheduck]

FutureEx*[T, E] = ref object of Future[T

chronos went this way indeed to allow a more gradual transition from non-raises to raises - and to have better compatibility between libraries that use and don't use raises lists - the syntax has changed slightly since the linked pr, the annotation now looks like this: proc f(): Future[...] {.async: (raises: [XYX]).} - chronos will automatically translate the future a FutureEx-like internal type so that "user code" doesn't have to change too much and to maintain the option of changing the raises representation should the language grow some helpful primitives for reasoning about raises effects the same way the raises keyword works.

Per our pre-raises porting guide, this usually is relatively pain-free, ie we can combine raises and non-raises-annotated libraries usually by doing work only on the raises-annotated side which ends up being pretty natural (that's where you're changing the code anyway).

The biggest problem arises when code uses async proc types for callbacks - in particular, if your api uses an async closure callback, enforcing no-raises on it significantly simplifies reasoning about correctness when calling the callback - many API become much more clean when error information is contained inside the callback instead of traversing the framework, but the lack of co/contravariance renders these assignments incompatible, even if in theory, they would work.

Finally, one of the biggest sources of bugs in asyncdispatch itself is uncaught exceptions that leak out of the event loop - before introducing raises in futures, we had to harden the core against those. The hardening of the core is what led to raises tracking being easier to introduce - code using chronos was already a little bit more exception-aware because of that.

[nitely]

The main issue I found with the chronos approach is it makes error tracking annotations viral. There is no inference. In a previous approach I tried rewriting the return type to FutureEx + doing raises inference in a later stage (typed macro), but the raises list is not available for async generic procs at that point. I also ran into issues of the type seq[Future] / seq[FutureEx] mismatch.

I agree callbacks need to not raise; For this PR latest approach, if we had an async keyword, I'd be able to detect the proc is async and not a closure that happens to return a future, but can raise anything because the future result is "user controlled"; that way we could rely on the error tracking inference for the async proc.

[arnetheduck]

There is no inference.

we started playing with that here: status-im/nim-chronos#327 - this is a little bit what I meant by adding language for raises computations - those would also include a raisesOf: codeBlock that gives you types that can be raised etc - I think rather than async being a keyword, there's a set of primitives that would add up to those kinds of features while being more flexible.