vegasq / gemini-dumper Goto Github PK
View Code? Open in Web Editor NEWgemini-dumper
gemini-dumper
Your Gemini downloader is breaking multiple Gemini sites because it's downloading pages TOO DAMN FAST! Please add a delay between requests per host. I've already blocked your IP address on my server, and I suspect others are doing so as well since I brought up the issue yesterday on the Gemini mailing list.
The code used to parse URLs is broken. The link gemini://gemini.conman.org/boston/2009/11/16.1#nf2009-11-16-1-2
is incorrectly parsed as gemini://gemini.conman.org/boston/2009/11/16.1%23nf2009-11-16-1-2
. RFC-2986 covers the syntax of URLs and how to properly resolve them. Or find a proper URL parser and use that.
Furthermore, the link mailto:[email protected]
does not appear anywhere on the site gemini://gemini.conman.org/
so it appears links from other sites are being leaked, or mixed up in some manner.
You are not properly parsing the Gemini links in your bot, specifically, relative links, which are allowed per the Gemini spec. Example: the link mailto:[email protected]
has been misparsed as gemini://gemini.conman.org/boston/2004/08/mailto:[email protected]
.
The fragment portion of a URL should not be sent as part of the Gemini request.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.