- Project uses colly framework
- Single threded only, no multithreading as of now
- Scraping every chapter takes somewhere between 2 and 3 minutes
- You can further optimize by updating variable
findLastChapter()
function uses to start search from. This will reduce wait time before scrape starts
- Move into
src
directory, inside the project, and run from there - Only use numbers as arguments
-
Single Chapter: Scrape specified chapter
go run . -single <Chapter>
-
Multi Chapter: Scrape multiple chapters at once
- Scrape every chapter
go run . -multi
- Scrape
<From>
till the endgo run . -multi <From>
- Scrape
<From>
to<to>
go run . -multi <From> <To>
- Scrape every chapter
-
Check if files already exist.
This will only check file names, not the actual content inside the files
go run . <Sinlge or Multi> -check
Check can be used on both
Single
andMulti
mode.As of now, you need some other mode first to check downloaded files
-
EPUB, compile all the downloaded txt files into epub file
go run . <Sinlge or Multi> -epub true
As of now, you need to run some other mode first to compile epub