How to filter ahrefs crawling bot about goatcounter HOT 4 CLOSED

These should already be recognized as a bot, based on the User-Agent header. Looking at goatcounter.com, this seems to work.

I wouldn't be surprised if this may be an artificial of the fix I sent you the other day 🤔 Does this happen for pageviews after 2023-05-17?

from goatcounter.

I'll keep an eye on it. Looking from the 18th-20th I see unknown browsers at 0%. I also have 0% for
April 18-20 so it looks like you may have the diagnosis right. I'll watch for the next week or so and if
I see few unknown browsers I'll close this issue.

Also one additional question.

I have a 30+% unknown Top Referrers. Does that category record only requests that do not have a referer (sic) header, or does it include a referer header without any value?

from goatcounter.

I have a 30+% unknown Top Referrers. Does that category record only requests that do not have a referer (sic) header, or does it include a referer header without any value?

It's for all cases where the value is empty: whether the header is omitted entirely or has a blank value.

I'm not sure if "header without a value" is something browsers actually send though. Typically if there is no value it will just omit the header entirely.

from goatcounter.

Looks like this was an artifact of the patch. Closing.

from goatcounter.