Fetch API: support redirection?

[peterhuene]

We've talked about mirroring (i.e. replicating) package logs from other registries, whereby the registry will keep a copy of a log streamed from upstream registries so it can directly serve the log to its clients.

For example, the wasi:http package originates from the registry.wasi.dev registry and the registry.neato.burrito registry is configured to mirror it by streaming events from registry.wasi.dev. With that, a client can fetch the log for wasi:http directly from registry.neato.burrito without taking to registry.wasi.dev.

In addition to mirroring, I think we should consider supporting redirection in the API. That is to say, a registry should be able to answer with "well, I can't serve you wasi:http directly, but you can find it over at registry.wasi.dev".

Having this would enable pretty cheap federation of registries even before we've designed a proper streaming API for replication. For example, I could configure my registry to redirect anything with a wasi namespace to registry.wasi.dev without much fuss. Clients that are configured to source packages solely from my registry now have access to WASI packages as well.

Right now the fetch API takes a batched set of packages to fetch and therefore an individual 3XX response can't be used.

To support this, perhaps we would need the fetch response to include a map of log ids to their redirected locations; the client would then turn around and delegate the relevant part of the fetch operation to those locations (this might require some finagling in the client storage implementation to ensure the per-registry storage works as intended).

Clients could be configured with a maximum redirection count and probably use a default of 1 with the expectation that a registry answers with the original location of the package.

What are folks thoughts on this?

[lann]

Makes sense to me, and could be useful beyond federation like for package sharding or serving log data via CDN. Might be good to have a higher redirect limit if those are compelling use cases.

[macovedj]

This is great. I have some open questions still around if/when content is mirrored along with logs.

For example, the wasi:http package originates from the registry.wasi.dev registry and the registry.neato.burrito registry is configured to mirror it by streaming events from registry.wasi.dev. With that, a client can fetch the log for wasi:http directly from registry.neato.burrito without taking to registry.wasi.dev.

Just to clarify, while the client gets logs from registry.neato.burrito about wasi:http, it still ends up fetching the content from registry.wasi.dev correct? At least in the general case where registry.neato.burrito doesn't mirror content.

Maybe the following would be nice. While each registry has it's map for redirection, the client could have one too. When it downloads a package from a registry that informs the client to redirect, the client could ask the user if they would like to update their config to match the redirection currently being recommended. Users could then easily see what redirection it's been informed to make historically, avoid redundant calls to ask about redirection, and also see which registries have historically made recommendations to the client that it it has already incorporated into its config, along with the query that motivated the current local client config.

[lann]

Good thought. The client wouldn't even need to store redirections per se; it can just store "this is the canonical registry for log id X".

[peterhuene]

Just to clarify, while the client gets logs from registry.neato.burrito about wasi:http, it still ends up fetching the content from registry.wasi.dev correct? At least in the general case where registry.neato.burrito doesn't mirror content.

In the mirroring case, the client would ask the mirroring registry for a particular release's content sources and attempt to fetch the content from the locations the registry returns. If the registry returns the original content locations as returned by registry.wasi.dev, then the client will fetch the content there. Instead, if the mirroring registry decided to also mirror the content, then it could return a set of locations for the content including its own in addition to the original or, alternatively, only the mirrored locations.

While each registry has it's map for redirection, the client could have one too.

I definitely think the client should keep a local mapping of the redirections from a particular registry so that it doesn't talk to the redirecting registry again for those particular packages and instead directly talks to where it should. That does imply the question of whether or not that cached mapping needs an expiration.

I don't know if we'd want to prompt the user in this case, as I can imagine this being a fairly common scenario for SDKs configured to talk to one particular registry (e.g. Fastly's, Fermyon's, etc.) and let it inform the client that those packages might exist, just not at the registry the SDK is configured with.

[macovedj]

I don't know if we'd want to prompt the user in this case, as I can imagine this being a fairly common scenario for SDKs configured to talk to one particular registry (e.g. Fastly's, Fermyon's, etc.) and let it inform the client that those packages might exist, just not at the registry the SDK is configured with.

Makes sense. The concern that motivated the proposal was that if we don't inform users that their config is going to be changed, then it may be harder for users to take control if/when two registries that they are fetching from fight over their config. I guess so long as the config is transparent to the user, they could manually modify it to override unwanted changes that registries try to make?

[macovedj]

In the mirroring case, the client would ask the mirroring registry for a particular release's content sources and attempt to fetch the content from the locations the registry returns. If the registry returns the original content locations as returned by registry.wasi.dev, then the client will fetch the content there. Instead, if the mirroring registry decided to also mirror the content, then it could return a set of locations for the content including its own in addition to the original or, alternatively, only the mirrored locations.

I think that we're getting to a point where more clarity around how content location is communicated to the client could be very helpful, (at least for me). I guess right now, there is no information in logs about urls used for fetching content, and then there is also the whole question about auth and restricted access.

I guess if you start with thinking about the simple case where we're only talking about public packages, and content download urls are in the logs, then does redirection need to happen? Essentially, if I understand correctly, neato.burrito instead of telling the client to go ask wasi.dev for a content fetching url, would tell the client "here's the url that wasi.dev would give you". I guess I'm not sure whether these urls would be in the logs themselves, or maybe mirroring could enable registries to communicate about this information privately, so that registries can speak on behalf of one another to clients about content locations that only package authors should be given, like publish urls. Maybe something like the latter would help keep logs small, and be helpful for not sharing urls for publishing new packages to various content stores?

I also think we've discussed mirroring being a mechanism for registries with more robust cloud resources to absorb some of the traffic for a registry with public packages that can't handle the demands its userbase. Maybe if registries tell clients how to fetch content without them needing to talk to the registry that is being mirrored, it could help with this goal as well?

I still very much like the idea of being able to ask registries how they're mapping namespaces to mirrored registries, and am trying to avoid changing the subject to content storage implementation details, but I think the topics are a bit related, and don't quite know how to separate them. If we want I could create another discussion for this topic?

[macovedj]

As a point of clarification, I'm generally talking about urls to get content from something like a bucket/blob storage. There are other ways we could inform users about where to get content. Right now I think we're only using local file systems for registry servers, and idk if we have any solutions in mind that are different from bucket/blob storage, since file systems probably won't scale the way we need.

[lann]

Content locations are intentionally decoupled from the log itself; the only cryptographic connection between them is the content digest in the log. Registries will generally advertise one or more (perhaps zero in edge cases like DMCA takedown) download URLs for each release and clients will verify downloaded artifacts against the in-log content digest.

[macovedj]

This makes sense to me. I think what I'm still looking for an answer to is the scope of registry mirroring. If registries can communicate amongst themselves about how to instruct clients to fetch content, then can a registry avoid telling a client to go look elsewhere, by just answering for the registry that it mirrors. I guess if it doesn't, then are we taking full advantage of the benefits that mirroring has to offer? If at scale, registries are using something like a blob store, does mirroring have both logs and fetch urls as part of it's scope?

[lann]

Yes, a mirroring registry can either reflect the upstream content location(s) and/or mirror the content itself into its own location(s). Some examples:

• bigcorp.com never wants to touch external hosting for its internal deployments, so they mirror logs and replace content locations with their own storage.bigcorp.com hosting.
• wasm-registry.ipfs.tech wants to promote the use of IPFS, so they mirror logs and replace content locations with mirrored ipfs:// locations
• wasm-paas.io wants to limit their egress costs but also wants to provide backup hosting for their customers, so they append registry-backup.wasm-paas.io/... to each content location list.

[esoterra]

Reasons bigcorp.com may not want to touch any other hosts include

• minimizing ingress/egress costs and network distance by using self-hosted sources,
• preventing their servers from reaching out into the public internet and thereby telling others what they're doing,
• running a registry in an air-gapped environment, and
• not becoming reliant on the availability or bandwidth of the upstream registry.