unicode fragments

[achristensen07]

Right now there is a note: "Unfortunately not using percent-encoding is intentional as implementations with majority market share exhibit this behavior."

I think we should change "Append c to url’s fragment." to at least "If url’s scheme is a special scheme, append c to url’s fragment. Otherwise, append the result of running UTF-8 percent encode with the simple encode set on c" or something similar.

Right now Firefox and Safari percent-encode non-ASCII characters in the fragment. Chrome percent-encodes non-ASCII characters in the fragment if the scheme is not a special scheme. Edge does not percent-encode any non-ASCII characters in the fragment.

This change is needed to continue to support percent-encoding non-ascii characters after a '#' in a data URL, which works in Safari, Chrome, and Firefox.

[achristensen07]

Ideally, I think we'd all just switch to percent encoding all non-ASCII characters in all fragments so the output of URL parsing is always ASCII. Would everyone be willing to make such a change?

[annevk]

@valenting? I think @dbaron also wanted us to always percent-encode, to keep URLs "ASCII safe" and easier to copy-and-paste (due to whitespace and such).

[valenting]

I support percent encoding all characters in the hash. It turns out percent encoded strings are much easier to manage and parse.

[annevk]

It seems HTML already decodes fragment identifiers in https://html.spec.whatwg.org/multipage/browsers.html#the-indicated-part-of-the-document so presumably that would keep working.

But I suspect there might be websites that depend on fragment not being encoded?

@achristensen07 did you verify that all APIs for URLs in WebKit return the URL with the fragment encoded as well? Or in order to have URLs with the fragment encoded would we need to add special casing to APIs?

[achristensen07]

In WebKit, the fragments are encoded/serialized as they are parsed, so there is no time in which there will be an unencoded fragment in a URL

[annevk]

@achristensen07 that's understood, but my question is if there's any APIs such as location.hash that might yield them as unencoded (essentially with the accessor doing its conversion).

[achristensen07]

I do not think we ever percent-decode fragments in WebKit. Going to the indicated part of the document with percent-encoded fragments or non-ASCII fragments which we percent encode doesn't work in WebKit right now, which we need to fix.

[dbaron]

I think @dbaron also wanted us to always percent-encode, to keep URLs "ASCII safe" and easier to copy-and-paste (due to whitespace and such).

In particular, the argument I made was in Mozilla bug 1148861 (comment 0 and comment 25).

[annevk]

So Gecko seems to be encoding too (although the address bar decodes). So that leaves Chrome and Edge. @foolip who's a good contact for URL parsing in Chrome?

[sleevi]

I'm still lurking watching these bugs, @annevk , at least for our low-level implementation (and how it relates to URL bar). When it get's to Blink side, @mikewest is good to poke just to make sure I don't stuff it all up :)

[foolip]

Thanks @sleevi. @sof has also poked a bit at the Blink-side URL.

[annevk]

@sleevi I guess the question is then whether you'd consider always encoding fragments. (Or is that already happening and is the Blink side doing magic in the API layer?)

[sleevi]

@annevk Yeah, I'm planning on looking into this today and trying to get an answer. I seem to recall that it's similar to Gecko, and so I suspect our URL substrate is decoding, and any encoding would need to be done at the Blink glue to that (the KURL and bindings side). On the upside, it makes it easier to change when it would only affect Chrome (our URL code is shared with a growing number of Google products due to Cronet embedding), the downside is it may require a more extensive audit of where we expose those URLs to a web-visible form.

I'm not hearing a request to change address bar behavior, but did I misunderstand the remarks re: FF? My gut is that it seems to make sense to change that too, to ensure authors aren't going to copy/paste?