This package contains a new URL type for Swift, written in Swift.

• The Getting Started guide contains an overview of how to use the WebURL type.
• The Full documentation contains a detailed information about at the API.

You may be interested in:

• This prototype port of async-http-client, which allows you to perform http(s) requests using WebURL, and shows how easy it can be to adopt in your library.

This URL type is compatible with the URL Living Standard developed by the WHATWG, meaning it more closely matches how modern web browsers behave. It has a fast and efficient implementation, and an API designed to meet the needs of Swift developers.

To use WebURL in a SwiftPM project, add the following line to the dependencies in your Package.swift file:

.package(url: "https://github.com/karwa/swift-url", from: "0.1.0"),

var url = WebURL("file:///usr/foo")!
url.pathComponents.removeLast()
url.pathComponents += ["lib", "swift"]
print(url) // file:///usr/lib/swift

var url = WebURL("http://example.com/currency/convert?amount=20&from=EUR&to=USD")!
print(url.formParams.amount) // "20"
url.formParams.to = "GBP"
print(url) // http://example.com/currency/convert?amount=20&from=EUR&to=GBP

The implementation is extensively tested, but the interfaces have not had time to stabilise.

While the package is in its pre-1.0 state, it may be necessary to make source-breaking changes. I'll do my best to keep these to a minimum, and any such changes will be accompanied by clear documentation explaining how to update your code.

I'd love to see this library adopted by as many libraries and applications as possible, so if there's anything I can add to make that easier, please file a GitHub issue or write a post on the Swift forums.

Aside from stabilising the API, the other priorities for v1.0 are:

1. file URL <-> file path conversion

   Having a port of async-http-client is a good start for handling http(s) requests, but file URLs also require attention.

   It would be great to add a way to create a file path from a file URL and vice-versa. This should be relatively straightforward; we can look to cross-platform browsers for a good idea of how to handle this. Windows is the trickiest case (UNC paths, etc), but since Microsoft Edge is now using Chromium, we can look to their implementation for guidance. It's also worth checking to see if WebKit or Firefox do anything different.

2. Converting to/from Foundation.URL.

   This is a delicate area and needs careful consideration of the use-cases we need to support. Broadly speaking, there are 2 ways to approach it:

   • Re-parsing the URL string.

     This is what WebKit does. The benefit is that it is straightforward to implement. The drawbacks are that Foundation refuses to accept a lot of URLs which the modern standards consider valid, so support could be limited. In at least one case that I know of, differences between the parsers have lead to exploitable security vulnerabilities (when conversion changes the URL's origin, which is why WebKit's conversion routine now includes a specific same-origin check).

     Something like this, with appropriate checks on the re-parsed result, may be acceptable as an MVP, but ideally we'd want something more robust with better support for non-http(s) URLs in non-browser contexts.

   • Re-writing the URL string based on Foundation.URL's components.

     This should ensure that the resulting URL contains semantically equivalent values for its username, password, hostname, path, query, etc., with the conversion procedure adding percent-encoding as necessary to smooth over differences in allowed characters (e.g. Foundation appears to refuse "{" or "}" in hostnames or query strings, while newer standards allow them, so we'd need to percent-encode those).

     The WebURL parser has been designed with half an eye on this; in theory we should be able to construct a ScannedRangesAndFlags over Foundation's URL string, using the range information from Foundation's parser, and URLWriter will take care of percent-encoding the components, simplifying the path, and assembling the components in to a URL string. That said, URLs are rarely so simple, and this process will need a very thorough examination and database of tests.

   Even after this is done, my intuition is that it would be unwise for developers to assume seamless conversions between Foundation.URL and WebURL. It should be okay to do it once at an API boundary - e.g. for an HTTP library built using WebURL to accept requests using Foundation.URL - but such libraries should convert to one URL type as soon as possible, and use that single type to provide all information used to make the request.

   As an example of the issues that may arise: if the conversion process adds percent-encoding, performing multiple conversions such as WebURL -> URL -> WebURL, or URL -> WebURL -> URL, will result in an object with the same type, but a different URL string (including a different hash value, and comparing as != to the starting URL). That would be a problem for developers who expect a response's .url property to be the same as the URL they made the request with. That's why it's better to stick to a single type conversion; when a developer sees that the response's .url property has a different type, there is more of a signal that the content may have changed slightly.

3. Benchmarking and optimizing setters, including modifications via pathComponents and formParams views.

Post-1.0:

4. Non-form-encoded query parameters.

   Like the formParams view, this would interpret the query string as a string of key-value pairs, but without assuming that the query should be form-encoded. Such an API was pitched for inclusion in the URL standard, but is not included since the key-value pair format was only ever codified by the form-encoding standard; its use for non-form-encoded content is just a popular convention.

   That said, it would likely be valuable to add library-level support to make this convention easier to work with.

5. Relative URLs.

   Have repeatedly been pitched for inclusion in the standard. Support can be emulated to some extent by using the thismessage: scheme reserved by IANA for this purpose, but it is still a little cumbersome, and is common enough outside of browser contexts to warrant its own type and independent test-suite. Implementation may be as simple as wrapping a WebURL with the thismessage: scheme, or as complex as the Saturn V rocket; it is really quite hard to tell, because URLs.

6. IDNA

   By far the biggest thing. See the FAQ for details.

I'm creating this library because I think that Swift is a great language, and it deserves a high-quality, modern library for handling URLs. It has taken a lot of time to get things to this stage, and there is an exciting roadmap ahead.

It demands a lot of careful study, a lot of patience, and a lot of motivation to bring something like this together. So if you (or the company you work for) benefit from this project, do consider donating to show your support and encourage future development. Maybe it saves you some time on your server instances, or saves you time chasing down weird bugs in your URL code.

In any case, thank you for stopping by and checking it out.

It may be surprising to learn that there isn't a single way to interpret URLs. There have been several attempts to create such a thing, beginning with the IETF documents RFC-1738 in 1994, and the revised version RFC-3986 in 2005.

Unfortunately, it's rare to find an application or URL library which completely abides by those specifications, and the specifications themselves contain ambiguitites which lead to divergent behaviour across implementations. Some of these issues were summarised in a draft working document by Sam Ruby and Larry Masinter. As the web continued to develop, the WHATWG and W3C required a new definition of "URL" which matched how browsers actually behaved. That effort eventually became the WHATWG's URL Living Standard.

The WHATWG is an industry association led by the major browser developers (currently, the steering committee consists of representatives from Apple, Google, Mozilla, and Microsoft), and there is high-level approval for their browsers to align with the standards developed by that group. The standards developed by the WHATWG are "living standards":

Despite the continuous maintenance, or maybe we should say as part of the continuing maintenance, a significant effort is placed on getting the standard and the implementations to converge — the parts of the standard that are mature and stable are not changed willy nilly. Maintenance means that the days where the standard are brought down from the mountain and remain forever locked, even if it turns out that all the browsers do something else, or even if it turns out that the standard left some detail out and the browsers all disagree on how to implement it, are gone. Instead, we now make sure to update the standard to be detailed enough that all the implementations (not just browsers, of course) can do the same thing. Instead of ignoring what the browsers do, we fix the standard to match what the browsers do. Instead of leaving the standard ambiguous, we fix the the standard to define how things work.

From WHATWG.org FAQ: What does "Living Standard" mean?

While the WHATWG has encountered criticism for being overly concerned with browsers over all other users of URLs (a criticism which, to be fair, is not entirely without merit), I've personally found the process to be remarkably open, with development occurring in the open on GitHub, and the opportunity for anybody to file issues or submit improvements via pull-requests. While their immediate priority is, of course, to unify browser behaviour, it's still the industry's best starting point to fix the issues previous standards have faced and develop a modern interpretation of URLs. Not only that, but it seems to me that any future URL standards will have to consider consistency with web browsers to have any realistic chance of describing how other applications should interpret them.

Not yet. It is important to note that IDNA is not (just) Punycode (a lot of people seem to mistake the two).

Actually supporting IDNA involves 2 main steps (well, actually more, but for this discussion we can pretend it's only 2):

1. Unicode normalization

IDNA also requires a unique flavour of unicode normalization and case-folding, defined by the Unicode Consortium. Part of this is just NFC normalization, but there are additional, domain-specific mapping tables as well (literally, specific to networking domains). The latest version of that mapping table can be found here.

2. Punycode

It is the result of this normalization procedure which is encoded to ASCII via Punycode. In this respect, Punycode acts just like percent-encoding: it takes a Unicode string in, and outputs an ASCII string which can be used to recover the original content.

Why not just use percent-encoding? Because percent-encoding is seriously inefficient; it turns every non-ASCII byte from the input in to 3 bytes from the output. DNS imposes limits on the maximum length of domain names (253 bytes total, 63 bytes per label), so a more space-efficient encoding was needed. Only the hostname uses IDNA, because it is the only part affected by this DNS restriction.

That Unicode normalization step is really crucial - unforuntately, Swift's standard library doesn't expose its Unicode algorithms or data tables at the moment, meaning the only viable way to implement this would be to ship our own copy of ICU or introduce a platform dependency on the system's version of ICU.

As it stands, this URL library doesn't contain any direct dependencies on system libraries, and I would dearly like to keep it that way. At the same time, it has long been recognised that the Swift standard library needs to provide access to these algorithms. So once this project has settled down a bit, my plan is to turn my attention towards the Swift standard library - at least to implement NFC and case-folding, and possibly even to expose IDNA as a standard-library feature, if the core team are amenable to that. I suspect that will largely be influenced by how well we can ensure that code which doesn't use IDNA doesn't pay the cost of loading those data tables. We'll see how it goes.

For the time being, we detect non-ASCII and Punycode domains, and the parser fails if it encounters them. Of those ~600 URL constructor tests in the WPT repository, we only fail 10, and all of them are because we refused to parse an address that would have required IDNA.

We will support it eventually, but it's just not practical at this very moment.