Data from 2018-12-15 contains duplicate tests for some sites about legacy.httparchive.org HOT 4 CLOSED

Thanks for reporting this. Can you give an example of the issue you're seeing (eg a query that produces duplicate results)?

from legacy.httparchive.org.

Hi Rick, working on the query at the moment… Using the MySQL schema something like the following should give you an indication.

select url, count(url) as ct from pages
where label = 'Dec 15 2018'
group by url
order by ct desc
having count(url) > 2 -- desktop and mobile

As I have a slightly modified schema I can't run this directly and I've no idea how to do this in big table.

from legacy.httparchive.org.

Oh, I thought I'd posted a list of URLs. Can't get the hang of GitHub's attachment thang. In any case seeing the same problem with the January results. Will try and post a list here. Was about 100,000 sites in for December 15th and affected both desktop and mobile crawls.
2019-01-duplicates.xlsx

from legacy.httparchive.org.

I think this is a similar issue to #156 where we had to restart the crawl for some reason. I think we have it under control now so this shouldn't affect future crawls.

from legacy.httparchive.org.