Compare:

mixinURLUtils(target, function getTheBase() { ... }, function updateSteps() { ... });

vs.

mixinURLUtils(target, {
  getTheBase() { ... },
  updateSteps() { ... }
});

new URL("http://domain/path", "");
new URL("http://domain/path", "/path2/");
new URL("http://domain/path", "//domain2/path2/");

If the the input URL is absolute, the base shouldn't matter (unless it's not a string and not undefined). It probably shouldn't even be parsed in such a case.

file:// URLs on Windows typically do not %-encode the colon after the drive letter. When building such URIs, this library drops the host of the base (I assume because it reads it as a port?)

Expected, Chrome:

new URL('/c:/baz/qux', 'file://host/foo/bar').href
"file://host/c:/baz/qux"

Actual, whatwg-url:

new URL('/c:/baz/qux', 'file://host/foo/bar').href
"file:///c:/baz/qux"

It actually happens even when you just set pathname to anything with a colon:

const uri = new URL('file://host/foo')
uri.host // 'host'
uri.pathname = '/c:/bar'
uri.host // ''

Code says

        // SPEC: says to use "get the base" algorithm,
        // but the base might've already been provided by the constructor.
        // Clarify!

I think the implementation here is not quite on-spec. If you provide a base URL to the constructor, then per https://url.spec.whatwg.org/#dom-url-urlurl-base, the internal getTheBase returns a (normalized, validated) version of the provided base.

So getURL() doesn't need a getTheBase argument. init() probably does, since getAnchor() or mixinAnchor() or whatever will need it.

I might be able to PR this in.

We should add a pointer in the readme saying this is up-to-date as of commit whatwg/url@29ffceb109 or similar.

I recently opened a Pull Request that adds tests to check whether the URL constructor would break when a URL object (instead of string) is passed in as the second argument: #42

However, It looks like the "built" lib/url.js file that we get when we npm install whatwg-url-compat does NOT pass these tests. Tests are not run against that file, and I get an the error like this:

var URL = require('whatwg-url-compat').createURLConstructor();

var b = new URL("https://developer.mozilla.org");
var c = new URL("en-US/docs", b);

This code snippet will throw the error:

node_modules/whatwg-url-compat/lib/url.js:455
  return url.replace(/^[\u0000-\u001F\u0020]+|[\u0000-\u001F\u0020]+$/g, "");
             ^

TypeError: url.replace is not a function
    at trimControlChars (/Users/wei/development/urbancompass/src/js/node_modules/whatwg-url-compat/lib/url.js:455:14)
    at new URLStateMachine (/Users/wei/development/urbancompass/src/js/node_modules/whatwg-url-compat/lib/url.js:480:18)
    at URL.init (/Users/wei/development/urbancompass/src/js/node_modules/whatwg-url-compat/lib/url.js:1324:18)
    at new URL (/Users/wei/development/urbancompass/src/js/node_modules/whatwg-url-compat/lib/url.js:1344:10)
    at Object.<anonymous> (/Users/wei/development/urbancompass/src/js/url-test.js:6:9)
    at Module._compile (module.js:435:26)
    at Object.Module._extensions..js (module.js:442:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:311:12)
    at Function.Module.runMain (module.js:467:10)

Currently non-enumerable and non-configurable, but they should be enumerable and configurable.

To fix this it's probably easiest to move from Object.defineProperty to just object literals with get x() {} and set x(v) {}.

should be typeof base === "function". More tamper-proof; works cross-realm; etc.

Having tests fail if upstream is updated is kind of sad; we acknowledge our current revision in the README and the tests should be for that revision.

Also we probably want the ability to point at PR branches.

I can probably work this in to my PR. Filing it in case I forget.

jsdom needs it: jsdom/jsdom#1265

Minor typo in source code

From what I understand the spec uses c (a "code point", which it writes as a string like "@"). We have c, a number, an c_str, the corresponding string. We pass them around together so that code which wants to do numeric comparison can use c, and code which wants to use string comparison can use c_str.

Buffers are such a Node thing... Besides, it's confusing that sometimes in the implementation buffer means a string and sometimes a Node Buffer object.

Uint8Array is probably what you want.

As far as I can tell there's no low-level access to parseError. This makes it impossible to implement a simple validator on top of this code. (It would be even better for that to have detailed messages of course, but a boolean is a start.)

Has URLUtils.searchParams been removed?

What is the best way to do this?

Unrelated: Why doesn't URL use the same terminology as RFC3986? (protocol -> scheme, hash -> fragment, search -> query, etc.)

using hostname setter leads to this backtrace:

TypeError: this[("parse" + this.state)] is not a function
at new URLStateMachine (/opt/node/node_modules/whatwg-url/lib/url-state-machine.js:477:43)
at Object.module.exports.basicURLParse (/opt/node/node_modules/whatwg-url/lib/url-state-machine.js:1148:15)
at URLImpl.hostname (/opt/node/node_modules/whatwg-url/lib/URL-impl.js:126:9)
at URL.set [as hostname] (/opt/node/node_modules/whatwg-url/lib/URL.js:132:25)

The problem lies in whatwg-url/lib/URL-impl.js +126 where stateOverride is set to "hostname" instead of "host name" as it is described in URLStateMachine prototype.

They can be buggy and there are no web platform tests for them. This is sad.

"browser": "lib/browser.js"

lib/browser.js:

module.exports = window.URL || window.webkitURL;

for browserify

Gives an error inside tr46. The problem is that this[urlSymbol].url.host is a number, not a string.

This is blocking Location in jsdom since one of the tests depends on origin working correctly.

At the moment failures which cause parsing to terminate are short-cutted by directly throwing an exception and where failure does not imply termination are handled by encoding failure in the return value.

However, to align more closely with the spec, they all should return a real failure value (probably symbol).

There already was a polyfill for the URL spec at Polymer/URL based off @annevk's original url. What's the reason you created another implementation? I'm not accusing you of NIH, just curious…

URLUtils is basically dead. Although url.spec.whatwg.org still needs to be updated for the URL constructor.

• Merge URLUtils into Location: whatwg/html@f0a7365
• Inline URLUtils for a and area elements: whatwg/html@bab6591

As you can see from this commits (or the corresponding spec sections) whatwg-url will need to expose things like basicURLParse(urlObject, stateOverride), or setUsername(urlObject, value), or serializeHost(host), or... Might be best to write the jsdom patch first then see what APIs it needs.

Test case:

"use strict";
const whatwgURL = require("whatwg-url-compat");
const URL = whatwgURL.createURLConstructor();

const url = new URL("/dom/nodes/CharacterData-appendData.html", "http://127.0.0.1:57100/");
console.log(url.host);
url.host = "http://w3c-test.org/";

Output:

$ iojs test.js
127.0.0.1:57100
C:\Users\Domenic\Dropbox\GitHub\tmpvar\jsdom\node_modules\whatwg-url-compat\lib\url.js:1337
  obj[updateStepsSymbol].call(obj, value);
                        ^

TypeError: Cannot read property 'call' of undefined
    at preUpdateSteps (C:\Users\Domenic\Dropbox\GitHub\tmpvar\jsdom\node_modules\whatwg-url-compat\lib\url.js:1337:25)
    at URL.host (C:\Users\Domenic\Dropbox\GitHub\tmpvar\jsdom\node_modules\whatwg-url-compat\lib\url.js:1187:5)
    at Object.<anonymous> (C:\Users\Domenic\Dropbox\GitHub\tmpvar\jsdom\test.js:7:10)
    at Module._compile (module.js:430:26)
    at Object.Module._extensions..js (module.js:448:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:471:10)
    at startup (node.js:117:18)
    at node.js:952:3

Anne's back from vacation and things are changing!!

whatwg/url@06b2e68...master

Changes to exact set of characters that are encoded, basically.

I think we should not do this without adding our own supplement to web-platform-tests for each case (in this repo).

This lib is written with ES6, so it doesn't get parsed correctly with browserify:

Parse error at -:3082,12
SyntaxError: Unexpected token: name (x)
Error

It'd be great if there were a browser version available to serve as a polyfill for IE.

... containing only a path and hash. Is this possible? I know that the browser implementation of URL does not.

var url = new URL("http://user:[email protected]/");
url.password = "";
console.log(url.href); // "http://user:@test.com/"

The expected output is "http://[email protected]/" and not "http://user:@test.com/".

There's no real name for this in the spec... in fact as I noted in jsdom/jsdom@b4c8e85#commitcomment-12481649, I'm not sure the spec actually calls for this to be used at all. Just an idea for a better name.

It seems like the spec doesn't contains any steps to validate IPv4 addresses, but the test cases in the web-platform-tests repository include some which validate proper handling of IPv4 addresses.

@annevk is that missing from the spec, or did I miss it somewhere?

Has it gained any traction in the industry or is it another proposal that could just get ignored?

Not necessary for v1. But I think that it would be really good, for both the implementation and the URL spec itself, to add some code coverage so we get some idea of how complete the test cases in web-platform-tests are, and can create new ones to target under-tested areas of the spec.

Some oddities show up while parsing file urls across all the envs I am looking at.

u = new URL('file://x#y')
u.href // 'file://x/#y'

The extra / shouldn't be there for this one.

console.log( JSON.stringify( new (require("whatwg-url").URL)("http://domain/") ) )

produces:

{}

I don't see any mention of "punycode" in the spec, and Chrome behaves differently:

new URL("http://faß.ExAmPlE").hostname
//-> fass.example

I don't know enough about text encoding to know if punycode is different from ascii/unicode

c_str is understandable, but there's several more which seem unnecessarily snake case.

URLUtils includes the "stringifier" IDL production which implies a toString method. The following should do:

toString() {
  return this.href
}

For some details see a098afd.

There's a mapping table we can import which should fix all of this if done properly. Maybe put this into its own module too?

Probably necessary for 1.0.0

I've always been a little afraid of that function, since IIUC it doesn't really follow any spec besides just "here is the behavior Brendan put in to ES1." Just wanted to check you were aware of its caveats and are sure it is being used correctly?

It creates, it doesn't really get. createURLConstructor? makeURLConstructor?

whatwg/url@7b40216

This seems important for when composing with HTML.

Try replacing all the indexOf calls.

About 83% of the whatwg-url as packaged on npm is consumed by the /coverage directory.
This is probably not intended.

To exclude that, one of the following should be done:

• it has to be either added to .npmignore
• .npmignore has to be deleted and entries should be merged with .gitignore (probably not an option, as you want to package ./lib)
• Use the files property in the package.json to explicitly specify the packaged files.

Ref: https://docs.npmjs.com/misc/developers#keeping-files-out-of-your-package

While playing with some jquery/ajax stuff in node.js, I think I’ve hit a bug in the whatwg-url module used by jsdom. It’s unable to parse certain valid URLs such as: "https://r3---sn-p5qlsnz6.googlevideo.com/" and throws "TypeError: Invalid URL". I think the problem is with the “r3---sn” part of the hostname.

jsdom version: 8.1.0
whatwg-url version: 1.0.1

Following node.js code reproduces the issue:

var url = "https://r3---sn-p5qlsnz6.googlevideo.com/";
parseURL = require("whatwg-url").parseURL;
var result = parseURL(url);

Output:

../node_modules/whatwg-url/lib/url-state-machine.js:1151
    throw new TypeError("Invalid URL");
    ^
TypeError: Invalid URL
    at Object.module.exports.basicURLParse (../node_modules/whatwg-url/lib/url-state-machine.js:1151:11)
    at module.exports.parseURL (../node_modules/whatwg-url/lib/url-state-machine.js:1189:25)
    at Object.<anonymous> (test.js:5:14)

The URL spec defines that origin may return a "globally unique identifier", which isn't described any further (see also whatwg/url#21).

In cases where the scheme isn't known, Chrome just returns scheme + "://". Should we just keep to Chrome here, @domenic?

http://:pass@domain/

throws an error. These parse fine, however:

http://user:@domain/
http://user@domain/
http://@domain/

whatwg/url#139 (comment)

The current bug we need to fix is that parsing /foo/bar against file:/path/to/docroot/index.html currently gives file:/foo/bar, whereas it should give file:///foo/bar.

In fact, just parsing file:/path/to/docroot/index.html should give back file:///path/to/docroot/index.html.

The complication here is that we have no tests for href serialization, which is horrible.

Related: whatwg/url#52 web-platform-tests/wpt#2038