Comments (10)
If I'm not mistaken it is the same repo as you created the issue. No need for a separate repo if you are able to incorporate the custom scoring inside searx-stats2.
Maybe @dalf will know more about this.
from searx-space.
We won't switch to SSLLabs (see #10 (comment)).
You may try to bump the issue that got opened on cryptcheck if you want, it seems like there was no response for 2 years: aeris/cryptcheck#30
Or if you find an alternative that support x25519 curve for the handshake and is as good as cryptcheck then we can consider to switch to it.
from searx-space.
So I believe that SSL Labs API with a custom scoring engine (similar to cryptcheck) would be accepted as a solution (since it comes only to their grading policy). Correct? @unixfox
from searx-space.
Why not, if you are willing to contribute about it, we can try to implement this custom scoring engine.
from searx-space.
Alright. It is going to be an interesting python experience for me. Even though I know a theory it has never been my language of choice - wish me a good luck :-). Also currently my schedule is a little busy so I am not going to give any ETA on the contribution. Any suggestion to which repo should I make a pull request with my change? Or should it be a brand new project?
from searx-space.
Yes, you can fork this repository and follow the instructions in README.md
The purpose is to change this cryptcheck_backend.py (and rename it).
More precisely, and readling https://github.com/TrullJ/ssllabs/blob/master/ssllabsscanner.py and https://github.com/ssllabs/ssllabs-scan/blob/master/ssllabs-api-docs-v3.md, for each host:
- start a new scan
- every 10 sec or so, check if the results are available.
In cryptcheck_backend.py
, the function analyse(host)
is called for each valid searx instance:
https://github.com/searx/searx-stats2/blob/04f62fb9b3f80c12de64566eab3521c7f622090c/searxstats/fetcher/cryptcheck_backend.py#L37-L38
Currently, there are 2 call in parallel, see limit=2
:
https://github.com/searx/searx-stats2/blob/04f62fb9b3f80c12de64566eab3521c7f622090c/searxstats/fetcher/cryptcheck_backend.py#L70
The output of analyse(host)
is a tuple: (rank, url)
.
You can make HTTP request:
async with new_client() as session:
response = await session.get(api_url, timeout=HTTP_REQUEST_TIMEOUT)
The new_client
function returns a HTTPX client. The API is very similar to requests (a lot of Python code use requests).
You can check for one or few URL from the command line:
python3 -m searxstats --cache /usr/local/searx-stats2/cache --upgrade-https-grade https://a.instance https://another.instance
Then you can check the output and / or the file html/data/instance.json
.
from searx-space.
@dalf That is a one, very detailed response. Thank you for that! It will help me for sure 😃. As I said, don't expect any pull requests yet. I still have a few things to work on.
from searx-space.
Hey @dalf
I finally feel confident enough in my Python skills to take care of this issue.
However, I have a question regarding the structure of the instance cache/output file (more detail on that later).
First of all, I have decided to use the https://github.com/devolo/ssllabs package as it seems to be the most actively maintained one.
This package requires me to update the httpx
package from 0.11 to 0.16 which breaks a few things, namely:
- the
httpx.backends.asyncio.AsyncioBackend
is no longer present as it seems that httpx has moved asyncio functionality to the exclusiveasyncio
package (related code: https://github.com/searx/searx-stats2/blob/master/searxstats/common/ssl_info.py#L77)
After spending some time with the code I found out that dropping this function would break the fetcher/basic.py
which I think is responsible for gathering Certificate
details (name, altname, country, org etc.) visible on the searx.space
website.
Now I want to fill that missing details with the new fetcher/ssllabs...
fetcher.
The only issue I am having right now is that the dictionary structure is not very clear to me and it would help a lot (+save time) if you could simply provide me the cache/output file with dumped dictionary structure. This way when implementing certificate details I would be sure not to break any rendering on the searx.space
website. I could generate the file myself but it would require me to get cryptcheck docker instance running and I think there would be many unnecessary steps to it.
I suppose it looks something like:
...
tls {
version: "..",
serialNumber: "..",
...
grade: "A+",
gradeUrl: "https://userendpoint"
}
...
The full file would be very handy in making sure I have not made any mistakes and would help me to visualize things.
And you, most likely, have the whole setup ready to go.
from searx-space.
Bump @dalf
from searx-space.
I'm sorry for long delay to answer.
First of all, I have decided to use the https://github.com/devolo/ssllabs package as it seems to be the most actively maintained one.
See #10 (comment)
Perhaps SSL labs has been updated since last year ?
If not, we will stick to cryptcheck.
About the SSL certificates, you are right.
It is possible with a hook, see encode/httpcore#107 (comment)
from searx-space.
Related Issues (20)
- Wrong country detection on multiple instances HOT 6
- feature request: add "accepts json requests" field HOT 1
- Do not classify the PING resource as customization HOT 2
- Add notice that SearX instances will get removed
- Remove code that was needed to support old searx instances HOT 1
- https://searx.space/ is not updated since 2023.6.5 HOT 5
- Back navigation is broken HOT 2
- yacy engine instances HOT 3
- New metric : instance age? HOT 1
- How to estimate latency to users in different countries (e.g. US)? HOT 1
- Search response time column: the test must fail when there are only wiki* engines in the result HOT 3
- instances that do not have google working as ranked lower HOT 1
- Provide API to search searxng instances HOT 2
- Google response time not set, but google engine works ..
- Add more filters HOT 2
- More/better explanations
- HTTP grade vs HTML HOT 1
- Add search options to Tor sections HOT 4
- Help button doesnt work in Tor section HOT 5
- Use mozilla/http-observatory instead of dalf/http-observatory
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from searx-space.