Expand desktop corpus to all ~4M desktop CrUX origins about httparchive.org HOT 4 CLOSED

1. All the old URLs are dead and gone now? (for desktop)

Of the 1.3M desktop URLs we've been testing for ~6 months, 1M of them are still in the corpus. We considered including the remaining 300k URLs for consistency across crawls but opted not to for simplicity.

(we = myself, @igrigorik @pmeenan @paulcalvano)

2. Presumably you're doing the same thing for mobile?

Yes. We plan to roll that out next month.

3. @brendankenny said there weren't a lot of https urls in the old HA url list.. Whats the balance in this new URL list?

This list is 60% HTTPS:

SELECT
  SUM(IF(STARTS_WITH(url, 'https'), 1, 0)) / COUNT(0) AS pct_https
FROM
  `httparchive.urls.2018_12_15_desktop`

4. We thought the 1.5M run was barely finishing in under 2 weeks. Is there new WPT capacity being added to handle this load?

Yeah, we used to run each page 3 times on desktop, 3 on mobile, and 1 on Lighthouse. As of December 1 we're doing it 1 time on all 3 settings. That affords us the space to increase the desktop corpus. To afford the mobile increase, we're going to reduce the crawl frequency from 15 days to monthly. I wrote up a short summary / sanity check in this doc.

from httparchive.org.

mysql> select count(0) from urlsdev;
+----------+
| count(0) |
+----------+
|  3880557 |
+----------+
1 row in set (0.00 sec)

mysql> select count(0) from urlsmobile;
+----------+
| count(0) |
+----------+
|  1294054 |
+----------+
1 row in set (0.44 sec)

from httparchive.org.

Random questions:

1. All the old URLs are dead and gone now? (for desktop)
2. Presumably you're doing the same thing for mobile?
3. @brendankenny said there weren't a lot of https urls in the old HA url list.. Whats the balance in this new URL list?
4. We thought the 1.5M run was barely finishing in under 2 weeks. Is there new WPT capacity being added to handle this load?

from httparchive.org.

Sounds great! Thanks for the answers.

from httparchive.org.