searxng / searx-space Goto Github PK

• For https://searx.be/ GIT_URL = 'unknown', the detected fork is https://github.com/paulgoio/searx
• For https://darmarit.org/searx/ GIT_URL is https://github.com/return42/searxng

In both cases, the detected fork is https://github.com/paulgoio/searx
It should be https://github.com/searxng/searxng

@dalf pushed a commit dalf@37beff9 that fixed the issue #8 but while reviewing the commit I found out that currently searx-stats2 display at a higher priority an instance that have a better Content Security Policy than an instance that doesn't contain any modified scripts (like tracking or ads) so more privacy friendly than an instance with modified scripts.

What should be better for the visitors: prioritize by default the privacy or the security?

Personally I think if #11 gets merged, privacy should be the priority because there are already good mechanisms in modern browsers to secure connections between a client and a server.
Whereas it's more common to have shady scripts built in a Searx instance than having external attackers trying to gather searches from visitors of an instance by injecting scripts into the HTML page.

Hi,

I have been upgrading our Searx instance à La Quadrature du Net (https://searx.laquadrature.net)
When looking at the https://searx.space/ it says that we have no IPv6.
However, we do have IPv6 and the server is configured to respond on IPv6.

`---> host searx.laquadrature.net
searx.laquadrature.net is an alias for tau.lqdn.fr.
tau.lqdn.fr has address 185.34.33.4
tau.lqdn.fr has IPv6 address 2a00:99a0:0:1000::4

And the relevant nginx configuration :

listen 185.34.33.4:443 ssl http2;
listen [2a00:99a0:0:1000::4]:443 ssl http2;
server_name searx.laquadrature.net;

I'm wondering what is happening, if it is a bug in the stats code, or because it is a CNAME ?

Currently, the expected branch name is "master":

It should be able to deal with other names like "main".
Related to searxng/searxng#263

I have recently started to host my searx instance and after doing some custom bot filtering I scored poorly on both TLS/CSP grades. It has been fixed by simply whitelisting some ips/user-agents. However days passed and TLS grade is not getting updated whereas CSP refreshed within ~1 day.

What leads me to believe that this is an issue:

• By checking the source I can see that you are requesting https://cryptcheck.fr/https/searx.monicz.pl.json file on which my instance does have an A+ grade.
• By checking other instances' TLS reports I can see that in some cases last scan date is a month old (for example: https://cryptcheck.fr/https/eulie.de last scanned at Mon, 10 Aug).

My instance:
https://searx.monicz.pl/

Hi,

The instances.json file contains data about removed engines.

Two examples are:

• faroo
• asksteem

On searx.space engines tab, either we remove the columns of engines that have been removed from searx master, or we should add a ⚠️ warning sign to the faroo and asksteem rows of searx instances that are still listing these removed engines from searx master:

And maybe for searx instances that are correctly updated and don't list some removed engines, we should precise with this kind of icon in the related rows:

I don't know which code has to be changed for that.

Best is to add these two signs and if some searx instances have not been updated in 90 days or so, they should be removed from the list of available searx instances.

This is derived from searx/searx#1972. I personally think that there should be a way to contact the maintainer(s) of a public instance (email for example). It is harder to trust this awesome service if there is no way to contact the maintainer(s).

My suggestion for this change is to tell the maintainer(s) to provide an email address at a minimum.

EDIT: I think I may have applied the labels wrong

If you want to see your searx instance added or removed from https://searx.space/ list, please add a comment here.

https://searx.space/ does not refresh results active instances, nibblehole.com - missing CSP grade. The search response time often shows incorrect data, suddenly 70% in red? I think the problem is with your network, maybe too much traffic on the server?

There are no IPv4 left in Europe since the end of November: https://www.ripe.net/publications/news/about-ripe-ncc-and-ripe/the-ripe-ncc-has-run-out-of-ipv4-addresses

We should rank Searx instances based on the fact that they aren't supporting IPv6 to promote the use of IPv6!

Endless stream of

###!!! [Parent][MessageChannel] Error: (msgtype=0xA10001,name=PVsync::Msg_Notify) Channel error: cannot send/recv

Perhaps related to #29

Currently incomplete code: https://github.com/dalf/searx-stats2/blob/master/searxstats/fetcher/dnssec.py

The python project dnsviz can check DNSSec (it is used by https://dnsviz.net)

Currently the project checks if an URL is searx instance using this regex:
https://github.com/dalf/searx-stats2/blob/634c30527e1c31dd25558780873408898b33ef16/searxstats/fetcher/basic.py#L15

Some people wants to customize this string, in this case the instance will appear in the "Error" tab (Searx instance not found).

It would be better to check the .../config URL.

Related to https://github.com/dalf/searx-stats2/issues/29#issuecomment-604428693

In the engine tab display the list of hidden engines (see PR https://github.com/dalf/searx-stats2/pull/38 )

is the website - https://searx.space/ still checking the instance status? it looks like the process to check the condition of instances stopped a few days ago.

In response to searx/searx#1853 (comment)

For now, the searx-stats2 project has collaborators: asciimoo and return42 (not sure how to open).

For now, I'm the one who host searx.space (Kimsufi host). Currently it hosts some other website for my personnal usage.

searx.space log contains: timestamp, method, url, proto, status, size (no IP, no user agent, no referrer). Most probably can remove completely, but I don't think there is a privacy problem here and it helps to know some information (bandwidth, ratio between the 200 and 304 http status code).

It runs the master branch of https://github.com/dalf/searx-stats2 (git pull && make docker-build are executed by me).
In /etc/cron.d/searx-stats2:

0 1,4,7,10,13,16,19,22 * * *   root cd /srv/searx-stats2 && make docker-run &> /tmp/searx-stats2.log || exit 0

It can be interesting to display the result of searx-stats2 from different location but IHMO the actual fetch should be done only once to avoid to hammer the different instances. Related to https://github.com/dalf/searx-stats2/issues/1

I thought about different things:

• use custom domains and GitHub Pages.
• run searx-stats2 in a VM, and always run the master branch (or a "searx-space" branch). Not sure if different root user is good idea (?).
• allow searx to download and display instances.json. Additional benefit: at the same time, each instance could ping searx-stats2 to say "hey I'm a public instance".
• spread instance.json using P2P networks.

It is difficult to combine everything:

• fetch HTTPS / CSP / HTML grades only once.
• get response time from only different location.
• AND avoid a the single point of failure / one central location
• AND #20 : how an ops can know some requests come from searx-stats2 if the check is run multiple times in different locations.

FYI, searx.me seems to be ban in China according to https://viewdns.info/ (it wasn't the case in 2016).

Unfortunately we're up against CloudFlare™️ and that is a problem, especially if we use Tor; so a proxy is needed. Some of the instances offer i.e. Morty, but many do not, and instances have been disappearing lately, or at least their Tor Hidden Services. So it would be nice to have a list of instances that offer a proxy.

Originally opened in: searx/searx#1852

Thanks. 🍻

searx.be is on networks from Oracle cloud and Hurricane Electric (tunnelbroker).
The reported country on searx.space is US but in fact it should be noted "DE" because the IP is from a server located in Germanay.

I used this tool for checking the real country location of an IP address: https://www.iplocation.net/ip-lookup. For IPv4 address of searx.be the reported country on this tool is Germany and for the IPv6 it's also Germany but only for DB-IP.

I know it's not a simple task to solve because I've read the code, it's using WHOIS for checking the country and Oracle cloud doesn't report the country in the WHOIS.

IP2Location offer a free Geo database which can be used locally and seems at least a bit more accurate than WHOIS, at least on the IPv4 address of searx.be: https://www.ip2location.com/demo

For reference: dalf/cryptcheck-backend#1 (comment)

searx-stats2 has some "Connection Timeout" error despite a reliable searx instance.

I have been able to reproduce the connection timeout with this code ( httpx==0.11.0 )

import httpx
async with httpx.AsyncClient() as c:
   r = await c.get('https://searx.be')

TimeoutError                              Traceback (most recent call last)
~/ve/lib/python3.7/site-packages/httpx/backends/asyncio.py in open_tcp_stream(self, hostname, port, ssl_context, timeout)
    198                     asyncio.open_connection(hostname, port, ssl=ssl_context),
--> 199                     timeout.connect_timeout,
    200                 )

/usr/lib/python3.7/asyncio/tasks.py in wait_for(fut, timeout, loop)
    422             await _cancel_and_wait(fut, loop=loop)
--> 423             raise futures.TimeoutError()
    424     finally:

TimeoutError: 

During handling of the above exception, another exception occurred:

ConnectTimeout                            Traceback (most recent call last)
<ipython-input-2-5d9ab2d17d71> in async-def-wrapper()

~/ve/lib/python3.7/site-packages/httpx/client.py in get(self, url, params, headers, cookies, auth, allow_redirects, timeout)
   1235             auth=auth,
   1236             allow_redirects=allow_redirects,
-> 1237             timeout=timeout,
   1238         )
   1239 

Note: the bug won't appear immediately after, I have to wait few minutes before the same error appears.

The last httpx version (0.13.3) seems to fix the problem.

Searx.space is only available on the clearnet. It has an option to display working Tor Searx instances maybe searx.space could also be reachable from an onion address to preserve the full privacy when searching for a searx instance hosted on Tor?

Search engines that have otherwise good tls, observatory ratings and so on are at the bottom of the list for not displaying search response times. Examples are: https://spot.ecloud.global/, https://start.paulgo.io/ and https://searx.feneas.org/. They all have in common that they use some sort of the simple theme as default. For my instance its just the simple theme with some css (https://github.com/paulgoio/searx). I also enabled /stats, which also did not help.
I can also see searx.space hitting filtron, but still no search response time:

2021-06-05 09:40:37 | [searx.space] 2021-06-05 07:40:37.752 51.15.252.168 GET start.paulgo.io/search?q=%21google+time "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0"
| 2021-06-05 09:40:37 | [searx.space] 2021-06-05 07:40:37.727 51.15.252.168 GET start.paulgo.io/?q=%21google+time "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0"
 | 2021-06-05 09:37:56 | [searx.space] 2021-06-05 07:37:56.011 51.15.252.168 GET start.paulgo.io/search?q=%21google+time "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0"
 | 2021-06-05 09:37:55 | [searx.space] 2021-06-05 07:37:55.987 51.15.252.168 GET start.paulgo.io/?q=%21google+time "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0"
 | 2021-06-05 09:35:26 | [searx.space] 2021-06-05 07:35:26.291 51.15.252.168 GET start.paulgo.io/search?q=%21wp+time "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0"
 | 2021-06-05 09:35:26 | [searx.space] 2021-06-05 07:35:26.265 51.15.252.168 GET start.paulgo.io/?q=%21wp+time "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0"
 | 2021-06-05 09:32:56 | [searx.space] 2021-06-05 07:32:56.609 51.15.252.168 GET start.paulgo.io/search?q=%21wp+time "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0"
 | 2021-06-05 09:32:56 | [searx.space] 2021-06-05 07:32:56.585 51.15.252.168 GET start.paulgo.io/?q=%21wp+time "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0"

Eyes Filter

Would make browsing https://searx.space/ a lot easier by country, if there was an easy way to toggle only instances hosted outside of 5 eyes, 9 eyes, and 14 eyes.
So I would suggest a basic dropdown near the country field, where you can just click 5, 9, or 14 eyes toggles, or select none to only see instances outside of the 14 eyes.

Searx.space is only available on the clearnet. It has an option to display working Tor Searx instances maybe searx.space could also be reachable from an onion address to preserve the full privacy when searching for a searx instance hosted on Tor?

I recently created gimmeasearx.
It would be great to mention it under "Meta-searx instances".

Giving a lower rating (and probably ranking lower) because a searx instance customized the default theme is actually quite bad.
A few examples that customized the default theme and are apart from that perfectly good Searx instances:

• https://search.stinpriza.org/ - Changed only the logo
• https://suche.elaon.de/ - Changed only the logo
• https://spot.ecloud.global/ - Change the logo and customized the CSS and JS

And I could think of another example that would give a bad rating: minifying on the fly all the assets that aren't minifyied like for example search_on_category_select.js.

Actually the idea of giving grade based on the fact that the HTML files aren't completely the same as the original source code of Searx is a good idea but it's reaching a field where bad and good intentions are mixed together. And it's in this field, using the automated way to distinguish the bad from the good is actually very hard to do.

asciimoo just introduced a new file called "translations.js" in the Searx source code: searx/searx@4ca0d8c

Since this commit it seems like searx.space flags all the Searx instances as Cjs that have the newer code with the "translations.js" file.

tls.imirhil.fr is deprecated: https://twitter.com/aeris22/status/1084576545308581890

As you can see there is a big difference between the old algorithm and the new one:

• old : https://tls.imirhil.fr/https/searx.pofilo.fr
• new : https://cryptcheck.fr/https/searx.pofilo.fr

Cloudflare and other MITM services can see the traffic between the visitor and the Searx instance.
This is a huge privacy issue.

Introducing a warning by for example making the certificate column red for the instances with a Cloudflare certificates is a good way to warn the visitor of a potential risk for his privacy.

Example of a potential modification of the interface:

Which issues block a first public release:

• https://github.com/dalf/searx-stats2/issues/9 : it will take some time to rewrite the code.
• https://github.com/dalf/searx-stats2/issues/10 : the https grade is not realiable for now. Being one the first sort criteria, I think it is important to fix this one.
• https://github.com/dalf/searx-stats2/issues/6 : solved by https://github.com/dalf/searx-stats2/pull/11
• https://github.com/dalf/searx-stats2/issues/7: move list of searx-instances from searx to searx-stats2
• Meta-searx instances: add a new file to the searx-instances project ?
• Should the Searx instances mailing list be referenced ?

This instance https://search.stinpriza.org/ is sometimes at the top of the list because the default engine sometimes instantly timeout and thus gives a very low response time:

Checking if the default engines are working is probably the best way to avoid having broken instances at the top of the list.
I remembered there were an instance like this a few months ago on stats.searx.xyz and it was always at the top of the list because it would instantly return no results.

TLS grade isn't displaying for a lot of instances even though most of them would have gotten A+.

Crypt check still displays correct rating though.

Check if either filtron or antibot-proxy are installed.

See https://github.com/dalf/searx-instances/issues/70

How to check:

• filtron: the default rules block the CURL user agent.
• antibot-proxy: ? not sure how ?
  • check for a cookie: should be searx with the default configuration ?
  • check for a reference to /searx.css (default configuration) ?

Currently, none of the Searx instances served from tor listed in the wiki are listed on the interface.

I think that the same problem has occurred

Originally posted by @canbeup in #60 (comment)

Started with commit 200c3a31 from PR 1791 the list of public searx instances moved to the documentation tree. If this PR is merged, the SEARX_INSTANCES_URL has to be changed to the new location.

TL;DR; all discussed here are ended in #12 and #13

Can you have an API which gives the table on searx.space online https sites in an array of array format or array of dictionary format?

It would be really helpful to find online searx instances with suitable characteristics. If you can add querying, it will be all the more better.

Thanks.

Being the one first criteria to sort the instance list, the tls grade measure has to be reliable, otherwise the order will be "blinking". Unfortunately, this is not the case right now using cryptcheck:

• no result will be returned if a server has an ipv6 miss-configuration.
• some other cases, I can't determine.

It is possible to run cryptcheck locally using docker and / or send pull request to the git repository.
But, it seems there is a consensus around ssllab, at least in the current instance list.

Here some information ssllab:

• ssllab API: https://github.com/ssllabs/ssllabs-scan/blob/master/ssllabs-api-docs-v3.md
• ssllab Terms and Conditions: https://www.ssllabs.com/downloads/Qualys_SSL_Labs_Terms_of_Use.pdf

Extract :

...
You are not allowed, without our express permission, to:

• use the API for commercial purposes;
• use the API on a public web site;
• publish any information received from us via the APIs without the owner’s express permission;
• distribute, proxy, or otherwise make the API available for access or use by any person or entity other than your authorized employees, including but not limited to acting as a service bureau or developing a competing product or service offering.

...

Currently, the response times are retrieved from only one server so from one location in the world.
Thus, some Searx instances could actually have a bad ranking because they are far away from the server that serve the statistics.

One way to fix that would be to test every Searx instances from multiple locations.
The problem is that renting a server running all the time in multiple locations in the world is expensive.

So I think the best way to test each Searx instance without paying anything would be to use the CDN of Zeit. They have quite a lot of servers in different locations of the world: https://zeit.co/docs/v2/network/regions-and-providers/
We would just have to implement the test into a lambda function.

The CSP column (*) uses the external service https://observatory.mozilla.org/ . For some reason, the results are not reliable.

(*) source code: https://github.com/dalf/searx-stats2/blob/master/searxstats/fetcher/mozillaobs.py

Most probably it would be better to embed the python code: https://github.com/mozilla/http-observatory

Moreover it would be possible to check instances not installed at the root domain ( https://.../searx ).

A JavaScript free version of the website would be great because in the Searx community it's pretty common to have users that doesn't enable JavaScript by default.
Moreover, Searx itself support searches without having JavaScript enabled and stats.searx.xyz display its results without having JavaScript enabled.

Description

All instances except one display 🟡 or ?.

Current way to have ✔️ and ❌

if your searx setup use docker, then see

• https://github.com/searx/searx-docker
• more precisely https://github.com/searx/searx-docker/blob/e4e3aa48f448c79873897fa216dd78cda9a43011/docker-compose.yaml#L91-L101
• https://github.com/searx/searx-checker
• https://hub.docker.com/r/searx/searx-checker

There is no documentation on how to install searx-checker if you don't use docker.
Basically you need to call every day (cron.daily) :

python3 checker/checker.py -o <somewhere>/status.json http://127.0.0.1:8888

Then map <your searx instance url>/status to <somewhere>/status.json

Example: https://a.searx.space/status

searx-checker should be embedded into searx

• searx/searx#1559

On searx.space, there is this sentence at the top of the front page:

See the About section to know how to edit this list or look for the meta searx instances

Before that the meta searx instances were listed first, then the online instances:
https://github.com/asciimoo/searx/wiki/Searx-instances/aea2f3feaeb3d5998a860b17c25874d2571d52e9

Thumb up: to answer yes.
Thumb down: to answer no.

In https://searx.space, the country shows PL, it should be FR.

$ whois 2001:41d0:8:4fd4::1
...
inet6num:       2001:41d0::/44
netname:        OVH-200141d00000
descr:          OVH
country:        FR
admin-c:        OK217-RIPE
tech-c:         OTC2-RIPE
mnt-by:         OVH-MNT
status:         AGGREGATED-BY-LIR
created:        2015-04-10T21:50:51Z
last-modified:  2015-04-10T21:50:51Z
source:         RIPE
assignment-size:64

Are instances in the "Offline & Error" category regularly scanned to see if they're working? If not, can you please verify if https://searx.bar is working and re-add it to the "Online HTTPS" list if so? Cryptcheck should be able to resolve the address now. That was the issue that led to Cryptcheck showing a timeout.

Cryptcheck results

This is a break down of issue #54 because it contained too many features at the same time.

Track the uptime of public instances. We could use uptime robot or another tool for that and then use their API for fetching the overall uptime. That's what Invidious does with their instances list: https://instances.invidio.us/

This is a follow up to #50.

So once again I am having an issue with my instance's TLS grade as apparently my ciphers are too modern. Talking about searx.monicz.pl here. And here is a result from cryptcheck itself: https://cryptcheck.fr/https/searx.monicz.pl

I believe this is related to me using x25519 curve for a handshake. And here is an issue posted on cryptcheck's repository: aeris/cryptcheck#30

+here are some extra TLS details from the ssllabs guys https://www.ssllabs.com/ssltest/analyze.html?d=searx.monicz.pl

A few words from me: x25519 is not an unusual curve to choose. It has been widely supported for a few good years now. From the ssllabs result you may find that my encryption is valid for all modern (and not) browsers like Chrome 69, Firefox 62.

My opinion is that cryptcheck is currently unable to process modern encryption thus an alternative should be found. Fortunately there are a few open-source projects which focus on bring ssllabs API to life. Learn more at https://www.ssllabs.com/projects/ssllabs-apis/index.html Some of them are developed in python so I believe that the implentation itself should not be a big of a hassle.

Ssllabs has been keeping up with the latest TLS improvements and vulnerabilities. I would say that it is a service of choice when it comes to testing your website's TLS configuration. And it also provides a TLS grading similar to cryptcheck's one.

Why for example https://search.seds.nl/ have a higher ranking than https://roteserver.de/searx/?
Shouldn't security be a higher factor than a better response time?

Previous on the wiki page, each user add:

• a link to her/his instance
• a link to check to https://www.ssllabs.com/ssltest/analyze.html?query=

So in a way (?), the user agree its server to be checked by ssllab.

Now, the workflow is different, I guess in https://github.com/dalf/searx-instances/blob/master/.github/ISSUE_TEMPLATE/add-instance.md should retrieve a user agreement to be checked by searx-stats2.

The PR searx/searx#1900 brings support for searx branding.

The .../config URL adds some new fields:

• brand
  • GIT_URL
  • DOC_URL

GIT_URL should be use to check the vanilla status.

Internal to do:

• get_repository: the directory parameter becomes a parent directory. The real directory becomes some like os.path.join(directory, sha256(url)).
• GIT_URL must be ready when external_ressources.py runs.
• If the HTML column is neither V nor E, perhaps the instance doesn't comply with the AGPL license (see dalf/searx-instances#23 )

Extended version:

• add a new html page (a new tab) in the output to list different forks.