38.107.216.10 down for 1 hour about sysadmin HOT 15 CLOSED

Relapse:
13 Aug 12:37: FIRING
13 Aug 13:42: RESOLVED

from sysadmin.

Citing #108:

@hellais: I wonder if maybe it's an mlab node. If that is the case we should probably switch over to using one of the ones that live on greenhost and is deployed by us.

from sysadmin.

A couple of things.

• It seems like this address is not the primary one the bouncer gives out.

• IRC we changed the mobile app to not use this http return json headers backend, but instead use the OONI run one (@sbs, @nuke can you confirm this?).

• On desktop if the test helper does not work, it will try using the next one or at least that should be how it works, if it doesn't work like that it's a bug in ooniprobe.

• That IP does in fact belong to MLAB infrastructure. See: https://www.measurementlab.net/infrastructure/. They sometime announce planned maintainence, but not always. I didn't see any notice of this particular event.

from sysadmin.

IRC we changed the mobile app to not use this http return json headers backend, but instead use the OONI run one (@sbs, @nuke can you confirm this?).

Yep, we use whatever the bouncer gives us.

from sysadmin.

Yep, we use whatever the bouncer gives us.

Ah, what happens if the collector and/or test helper you picked doesn't work? Do you move onto the next one the bouncer gave you or just fail?

from sysadmin.

Ah, what happens if the collector and/or test helper you picked doesn't work? Do you move onto the next one the bouncer gave you or just fail?

Fail. (Is it the case that the bouncer returns more than one or is the question just speculative?)

from sysadmin.

FTR: from a regress test run on Travis, I have actually seen that what the bouncer sends us is:

{"net-tests": [
    {"collector-alternate": [
        {"type": "https", "address": "https://c.collector.ooni.io:443"},
        {"front": "a0.awsstatic.com", "type": "cloudfront", "address": "https://das0y2z2ribx3.cloudfront.net"}
    ],
    "version": "0.0.1",
    "name": "ooni_test",
    "test-helpers": {},
    "test-helpers-alternate": {},
    "collector": "httpo://42q7ug46dspcsvkw.onion", "input-hashes": null
}]}

from sysadmin.

FTR: from a regress test run on Travis, I have actually seen that what the bouncer sends us is:

but this is for a test without any test helpers.

Fail. (Is it the case that the bouncer returns more than one or is the question just speculative?)

The bouncer currently returns two different collectors/test-helper sets.

from sysadmin.

but this is for a test without any test helpers.

I think you say so because there are no helpers. But does this make any difference in regards to the collector, i.e. is the types of collectors returned dependent on the test name?

The bouncer currently returns two different collectors/test-helper sets.

I think you say so because you count the https collector/test-helper as one and the cloudfronted collector/test-helper as another, right? This is consistent with what I see in the JSON:

For http_invalid_request_line:

{
    "net-tests": [
        {
            "collector": "httpo://ganriuwqt5vgbgnj.onion",
            "collector-alternate": [
                {
                    "address": "https://b.collector.ooni.io:443",
                    "type": "https"
                },
                {
                    "address": "https://das0y2z2ribx3.cloudfront.net",
                    "front": "a0.awsstatic.com",
                    "type": "cloudfront"
                }
            ],
            "input-hashes": null,
            "name": "http_invalid_request_line",
            "test-helpers": {
                "tcp-echo": "37.218.247.110"
            },
            "test-helpers-alternate": {},
            "version": "0.0.2"
        }
    ]
}

For web connectivity:

{
    "net-tests": [
        {
            "collector": "httpo://ganriuwqt5vgbgnj.onion",
            "collector-alternate": [
                {
                    "address": "https://b.collector.ooni.io:443",
                    "type": "https"
                },
                {
                    "address": "https://das0y2z2ribx3.cloudfront.net",
                    "front": "a0.awsstatic.com",
                    "type": "cloudfront"
                }
            ],
            "input-hashes": null,
            "name": "web_connectivity",
            "test-helpers": {
                "web-connectivity": "httpo://2lzg3f4r3eapfi6j.onion"
            },
            "test-helpers-alternate": {
                "web-connectivity": [
                    {
                        "address": "https://b.web-connectivity.th.ooni.io:443",
                        "type": "https"
                    },
                    {
                        "address": "https://d2vt18apel48hw.cloudfront.net",
                        "front": "a0.awsstatic.com",
                        "type": "cloudfront"
                    }
                ]
            },
            "version": "0.0.1"
        }
    ]
}

So, if what you mean is that we should try with cloudfronted if https fails, then I think we agree in this respect. However, we never tested cloudfronted with MK and for this reason we really never enabled code to do cloudfronted even though all the required pieces should probably be there.

If instead you mean that one should receive more than one set of https collectors or test-helpers, I am actually not seeing that in the logs. But, in principle, it would probably be sensible to be prepared for this case and perhaps also retry in such case.

One thing that I was wondering is: is there any guarantee from the bouncer side that the returned https and cloudfronted collector are disjoint? With this I mean: one thing is if the https collector fails because it is censored, then it makes sense to try cloudfronted. Another thing is if it fails because the deployment went wrong. In such case, are we going to submit through cloudfronted to the same non-correctly deployed backend or are they disjoint?

from sysadmin.

Relapse:
23 Aug 11:08: FIRING
23 Aug 12:13: RESOLVED

from sysadmin.

Relapse:
28 Aug 12:36: FIRING
28 Aug 15:51: RESOLVED

from sysadmin.

08 Sep 12:45: FIRING
08 Sep 13:15: RESOLVED

from sysadmin.

13 Sep 14:56: FIRING
13 Sep 15:21: RESOLVED

from sysadmin.

23 Oct, 17:16: FIRING
23 Oct, 17:46: RESOLVED

from sysadmin.

ooni/backend#343

from sysadmin.