vegasq / gemini-dumper Goto Github PK

Your Gemini downloader is breaking multiple Gemini sites because it's downloading pages TOO DAMN FAST! Please add a delay between requests per host. I've already blocked your IP address on my server, and I suspect others are doing so as well since I brought up the issue yesterday on the Gemini mailing list.

The code used to parse URLs is broken. The link gemini://gemini.conman.org/boston/2009/11/16.1#nf2009-11-16-1-2 is incorrectly parsed as gemini://gemini.conman.org/boston/2009/11/16.1%23nf2009-11-16-1-2. RFC-2986 covers the syntax of URLs and how to properly resolve them. Or find a proper URL parser and use that.

Furthermore, the link mailto:[email protected] does not appear anywhere on the site gemini://gemini.conman.org/ so it appears links from other sites are being leaked, or mixed up in some manner.

You are not properly parsing the Gemini links in your bot, specifically, relative links, which are allowed per the Gemini spec. Example: the link mailto:[email protected] has been misparsed as gemini://gemini.conman.org/boston/2004/08/mailto:[email protected].

The fragment portion of a URL should not be sent as part of the Gemini request.