Bouncer giving out down test-helpers for ~16 hours about sysadmin HOT 5 CLOSED

Another option is to take RUM route: count volume or percentage of erratic measurements coming to the collector.

from sysadmin.

So it turned out that actually the httpth was also wrongly configured on the bouncer and we were giving out an address of a down service for that until ~21:00 2019-10-26 UTC.

from sysadmin.

For people accessing this incident issue from the internet, the incident has been fully resolved since Saturday evening.

The incident affects measurements from the following time period: 2019-10-24 16:00 UTC - 2019-10-26 23:00 UTC.

We keep the issue ticket open until we have taken all the steps we consider necessary to prevent it re-occurring in the future.

from sysadmin.

The way to check if the measurement you are looking at is a false positive is:

1. Look at the test_keys->failure key and seeing if it says connection_refused (it should not say this)
2. Checking if the test_helpers->backend says something other than http://37.218.247.94:80 or 37.218.247.93

from sysadmin.

ooni/backend#343

from sysadmin.