searxng / filtron Goto Github PK

I have a problem where if I search a string that is longer than 120 character it gets rejected by filtron.

Example:
search?q=ayIWuCsmIV0OiwMvQ7h5bTuMP2mxzhfVZJX7ulmeyIAYfOMeYUwa0J4VNU3rDwZBncYwlPYYg7dIaFq8pPQnESYuZsTbe4L4biuqHqJGXnwO896V2MObqeeZ works correctly.
search?q=ayIWuCsmIV0OiwMvQ7h5bTuMP2mxzhfVZJX7ulmeyIAYfOMeYUwa0J4VNU3rDwZBncYwlPYYg7dIaFq8pPQnESYuZsTbe4L4biuqHqJGXnwO896V2MObqeeZa gets rejected with error Too big request header

I can't reproduce this on my test server with identical filtron and SearXNG setup.
There is also nothing getting logged. When I inspect filtron with debugging enabled I can see every other request but as soon as this string length limit is reached the request won't even be logged.

What should I do now?

A prototype to replace filtron with Python code & redis:
https://gist.github.com/dalf/705ee88381a1b3254b5dce3052ea20e5

• redis stores only HMAC of the informations:
  • aggregation key is HMAC(secret_key, key1 || key2 || key3...)
  • so only the Python code has access to the data, Redis never knows the real data.
• the cost is less than one 1ms per URL on my laptop, YMMV.
• the agregation (IP and UserAgent) really works as intended (filtron actually does two different aggregations).

Right now searxng-docker uses dalf/filtron:latest as the image for filtron. Since we have filtron in the repo now we should setup github actions to bould images and push them to the searxng org.

Furthermore I think we should include the default rules.json from searxng-docker since in most cases people are just mounting that file inside their filtron container anyway. This way the file stays up to date, the container does not require mounting (and before that creating) a file and easier config for new maintainers. If anyway wants to change the rules.json they can just mount a custom one into the container at the location of the default one.

Backport of https://github.com/paulgoio/searxng/blob/main/src/rules.json

Sandbox:

• https://sandbox.al-f.net
• https://sandbox.al-f.net/rules

It seems to work.

ping @mrpaulblack