URL Normalization in query parameters

[EdisonAgudelo]

Hi everyone.

I'm new to Rio framework, and I'm really amazed with all the features it has. At the moment, there is something that I don't really understand...

Why URL is normalized by lowering case? I mean, there are some parts of the URL that are case-sensitive like query parameters, and in my case I could not find any way to obtain them without this normalization.

Have you considered this scenario? Or maybe, am I missed something in docs?

Thanks in advance.

[Aran-Fey]

Good point, query parameters definitely shouldn't be lower-cased. I can't remember if we had a strong reason for lowercasing the URL, maybe we should get rid of this normalization altogether. I'll talk it over with the team.

[mad-moo]

The normalization is in place because URLs aren't supposed to be case sensitive, and this is an easy way to achieve it. While we could internally normalize strings just before comparison, it's also common for user code to interact with the URL directly, and normalizing it fixes the problems for everyone, not just Rio.

The spec is actually very clear about this: We should lowercase the protocol and host, and nothing else: https://www.rfc-editor.org/rfc/rfc3986#page-11

That seems contrary to what I'd at least expect though. I'd also keep the path normalized of that's okay with everyone

[EdisonAgudelo]

Thanks for the quick response.

For my use case, path normalization does not represent an issue.... but, taking into account that this is really powerful framework and is growing, probably is better to be stick with the spec:

6.2.2.1. Case Normalization

For all URIs, the hexadecimal digits within a percent-encoding
triplet (e.g., "%3a" versus "%3A") are case-insensitive and therefore
should be normalized to use uppercase letters for the digits A-F

When a URI uses components of the generic syntax, the component
syntax equivalence rules always apply; namely, that the scheme and
host are case-insensitive and therefore should be normalized to
lowercase. For example, the URI HTTP://www.EXAMPLE.com/ is
equivalent to http://www.example.com/. The other generic syntax
components are assumed to be case-sensitive unless specifically
defined otherwise by the scheme (see Section 6.2.3).

Also, I have seen a feature request for dynamic route... So probably this will also require being case-sensitive

[Aran-Fey]

I don't see a reason to mess with the URL at all tbh. Seems like a "feature" that causes way more problems than it solves.

[mad-moo]

This was introduced precisely because it has solved a bunch of issues 🤔

[Aran-Fey]

Like?

[mad-moo]

The current URL is frequently used in comparisons. Navbars highlight the correct button based on it. Guards use it to determine whether a user should be permitted to access a page. That's why it's exposed to the user after all.

Dropping normalization would require all users that access the URL to be hyperaware of its casing and constantly add .lower() to their code. This is error prone and indeed Rio itself has had so many bugs because even we couldn't keep it straight, that normalization was finally introduced. It seems silly to me to move that burden to the user, especially since we're targeting simplicity.

On the flip side, what issues does correct normalization introduce? You'd be somewhat limited in choice of component parameter names, as they would all have to be lowercase. But that is exactly what Python convention tells you to do anyway, so that hardly seems like a problem.

[Aran-Fey]

I don't understand what causes issues with casing. How do you accidentally get my-website.com/Foo instead of my-website.com/foo?

[Aran-Fey]

Dropping normalization would require all users that access the URL to be hyperaware of its casing and constantly add .lower() to their code.

That's not true, it would require you to leave the URL as it is. For what reason would you lowercase it?

[mad-moo]

That makes no sense. Can you explain?

[Aran-Fey]

Huh? Explain what? I see no reason to lowercase URLs just like I see no reason to lowercase other strings, like the name of the app or the path of an asset or the text in a label. Why would you do that? What is the purpose?

[mad-moo]

So https://rio.dev/Docs should display a 404 because the "real" URL is https:/rio.dev/docs ? Seems insane to me

[Aran-Fey]

I'm completely fine with that. And considering that query parameters and url parameters for dynamic routes shouldn't be normalized anyway, I'd rather not normalize anything at all, rather than normalize some things and not others.

[mad-moo]

I disagree. The case sensitivity has already been an issue far too often and I can't imagine that being expected behavior for most developers

[Aran-Fey]

When was it an issue? I don't know anything about that.

[mad-moo]

I don't have a date for you. But we had people go to URLs where the path didn't match and getting error pages. So we fixed that by making the path insensitive, and suddenly had new issues because the navbar wouldn't display the current page correctly. I believe this was also browser specific, because some browser decided to normalize but not others. Not sure anymore. (May also not have been in the path, idk)

Dropping normalization means forcing these issues on every new Rio developer.

No normalization isn't an option anyway because the host MUST be normalized. The case of "don't normalize anything because that's easy to understand" doesn't exist.

We could conceivably treat matches as case insensitive UNLESS there is a clash between two pages, but now we're adding complexity for - let's be honest - a useless feature.

Our goal has always been to keep things simple for developers and that's what we should keep doing IMO.

[Aran-Fey]

No normalization isn't an option anyway because the host MUST be normalized.

But is rio responsible for that? Surely that's the browser's responsibility, if anything.

[mad-moo]

Why? The spec clearly says it's case insensitive. It would be perfectly fine for the browser to capitalize every second letter if it wanted to do that

[Aran-Fey]

Ok, fine. We don't have a responsibility to normalize it, though. Being case insensitive is not the same thing as being all lowercase anyway. I'd love to understand how people ended up having issues with casing, because to me it seems like we're messing with things we shouldn't.

[mad-moo]

I think it's just a matter of not being careful, because URLs aren't normally portrayed as case sensitive

[EdisonAgudelo]

From all comments above, I see that case normalization was introduced to prevent common errors from user side (probably no developer), but there are use cases that case-sensitive is expected.

What about a merged behavior?

I mean make comparison (internally) in case-insensitive except if there is a collision... leaving URL without modifications
Or maybe to have an extra attribute like original_url IDK.

IMO, I would like to be as most standard as possible... and taking into account that this is a FastAPI based app, they also have case-sensitive paths and I would expect that from Rio too.

[iwr-redmond]

I like the idea of supporting a case-insensitive default unless there is a conflict caused by the user specifying two URLs whose sole differentiation in a difference in the case of certain characters. However, if this makes matters unnecessarily complex it might be best to stick to case-insensitive as the sole supported standard.