TLS grade having a hard time to update about searx-space HOT 10 OPEN

If I'm not mistaken it is the same repo as you created the issue. No need for a separate repo if you are able to incorporate the custom scoring inside searx-stats2.

Maybe @dalf will know more about this.

from searx-space.

We won't switch to SSLLabs (see #10 (comment)).

You may try to bump the issue that got opened on cryptcheck if you want, it seems like there was no response for 2 years: aeris/cryptcheck#30

Or if you find an alternative that support x25519 curve for the handshake and is as good as cryptcheck then we can consider to switch to it.

from searx-space.

So I believe that SSL Labs API with a custom scoring engine (similar to cryptcheck) would be accepted as a solution (since it comes only to their grading policy). Correct? @unixfox

from searx-space.

Why not, if you are willing to contribute about it, we can try to implement this custom scoring engine.

from searx-space.

Alright. It is going to be an interesting python experience for me. Even though I know a theory it has never been my language of choice - wish me a good luck :-). Also currently my schedule is a little busy so I am not going to give any ETA on the contribution. Any suggestion to which repo should I make a pull request with my change? Or should it be a brand new project?

from searx-space.

Yes, you can fork this repository and follow the instructions in README.md

The purpose is to change this cryptcheck_backend.py (and rename it).

More precisely, and readling https://github.com/TrullJ/ssllabs/blob/master/ssllabsscanner.py and https://github.com/ssllabs/ssllabs-scan/blob/master/ssllabs-api-docs-v3.md, for each host:

• start a new scan
• every 10 sec or so, check if the results are available.

In cryptcheck_backend.py, the function analyse(host) is called for each valid searx instance:
https://github.com/searx/searx-stats2/blob/04f62fb9b3f80c12de64566eab3521c7f622090c/searxstats/fetcher/cryptcheck_backend.py#L37-L38

Currently, there are 2 call in parallel, see limit=2:
https://github.com/searx/searx-stats2/blob/04f62fb9b3f80c12de64566eab3521c7f622090c/searxstats/fetcher/cryptcheck_backend.py#L70

The output of analyse(host) is a tuple: (rank, url).

You can make HTTP request:

async with new_client() as session:
  response = await session.get(api_url, timeout=HTTP_REQUEST_TIMEOUT)

The new_client function returns a HTTPX client. The API is very similar to requests (a lot of Python code use requests).

You can check for one or few URL from the command line:
python3 -m searxstats --cache /usr/local/searx-stats2/cache --upgrade-https-grade https://a.instance https://another.instance

Then you can check the output and / or the file html/data/instance.json.

from searx-space.

@dalf That is a one, very detailed response. Thank you for that! It will help me for sure 😃. As I said, don't expect any pull requests yet. I still have a few things to work on.

from searx-space.

Hey @dalf

I finally feel confident enough in my Python skills to take care of this issue.
However, I have a question regarding the structure of the instance cache/output file (more detail on that later).

First of all, I have decided to use the https://github.com/devolo/ssllabs package as it seems to be the most actively maintained one.

This package requires me to update the httpx package from 0.11 to 0.16 which breaks a few things, namely:

• the httpx.backends.asyncio.AsyncioBackend is no longer present as it seems that httpx has moved asyncio functionality to the exclusive asyncio package (related code: https://github.com/searx/searx-stats2/blob/master/searxstats/common/ssl_info.py#L77)

After spending some time with the code I found out that dropping this function would break the fetcher/basic.py which I think is responsible for gathering Certificate details (name, altname, country, org etc.) visible on the searx.space website.

Now I want to fill that missing details with the new fetcher/ssllabs... fetcher.

The only issue I am having right now is that the dictionary structure is not very clear to me and it would help a lot (+save time) if you could simply provide me the cache/output file with dumped dictionary structure. This way when implementing certificate details I would be sure not to break any rendering on the searx.space website. I could generate the file myself but it would require me to get cryptcheck docker instance running and I think there would be many unnecessary steps to it.

I suppose it looks something like:

...
tls {
  version: "..",
  serialNumber: "..",
  ...
  grade: "A+",
  gradeUrl: "https://userendpoint"
}
...

The full file would be very handy in making sure I have not made any mistakes and would help me to visualize things.
And you, most likely, have the whole setup ready to go.

from searx-space.

Bump @dalf

from searx-space.

I'm sorry for long delay to answer.

First of all, I have decided to use the https://github.com/devolo/ssllabs package as it seems to be the most actively maintained one.

See #10 (comment)
Perhaps SSL labs has been updated since last year ?
If not, we will stick to cryptcheck.

About the SSL certificates, you are right.
It is possible with a hook, see encode/httpcore#107 (comment)

from searx-space.