An app to download novels from online sources and generate e-books.

Discord: https://discord.gg/wMECG2Q

Telegram: https://t.me/epub_smelter_bot

• Lightnovel Crawler
  • Table of contents
  • Installation
    • Standalone Bundle (Windows, Linux)
    • PIP (Windows, Mac, and Linux)
    • Termux (Android)
    • Chatbots
      • Discord
      • Telegram
    • Heroku Deployment
  • Running from source
  • Running the Bots
  • General Usage
    • Available options
    • Example Usage
  • Development
    • Adding new source
    • Adding new Bot
    • Supported sources
    • Rejected sources
    • Supported output formats
    • Supported bots

This application uses Calibre to convert ebooks.
Install it from https://calibre-ebook.com/download
Without it, you will only get output in epub, text, and web formats.

⏬ Windows: lncrawl.exe ~ 25MB

In Windows 8, 10 or later versions, it might say that lncrawl.exe is not safe to dowload or execute. You should bypass/ignore this security check to execute this program.

⏬ Linux: lncrawl ~ 30MB

It is recommended to install via pip if you are on Linux

⏬ To get older versions visit the Releases page

📦 A python package named lightnovel-crawler is available at pypi.

Make sure you have installed Python v3.6 or higher and have pip enabled. Visit these links to install python with pip in Windows, Linux and MacOS. Feel free to ask on the Discord server if you are stuck.

To install this app or to update installed one via pip, just run:

$ pip install --user -U lightnovel-crawler

In some cases you have to use python3 -m pip or pip3 or python -m pip. And you do not need --user option, if you are running from root.

Next, open your terminal and enter:

$ lightnovel-crawler

# Or, a shortcut:
$ lncrawl

To view extra logs, use: lncrawl -lll

There is no official support to run python in mobile devices. It is not guaranteed that the app will run smoothly in all devices. It is recommended to use the bots on either Discord or Telegram if you are on mobile.

📱 Using Termux, you can run this app in your android phones too. Follow this instructions:

• Install Termux from playstore.
• Open the app and run these commands one by one:
  • apt update && apt upgrade
  • termux-setup-storage
  • pkg install ndk-sysroot make python zlib clang
  • pkg install libxml2 libxslt libiconv libcrypt libffi zlib libjpeg-turbo
  • pip install -U lightnovel-crawler to install the latest version of this app.
• Now exit the console and relaunch it.
• Type cd ~/storage/downloads to store novels there.
• Type lncrawl to start.
• You can navigate up using Volume UP + W and down using Volume UP + S.

When there is a new update available, you can install it just by running pip install -U lightnovel-crawler. You will not have to run all the above commands again.

Join our server: https://discord.gg/7A5Hktx

Or, visit this link to install discord bot to your own server: https://discordapp.com/oauth2/authorize?client_id=537526751170002946&permissions=51264&scope=bot

Visit this link to get started with the telegram bot: https://t.me/epub_smelter_bot

Send !help to open the bot help message.

Simply fill out the environment variables and you get a running instance.

• First clone the repository:

$ git clone https://github.com/dipu-bd/lightnovel-crawler

• Open command prompt inside of the project folder and install requirements:

$ pip install --user -r requirements.txt

• Run the program (use python v3.6 or higher):

$ python __main__.py

# Or, in short,
$ python .

There are two chatbots available at this moment: Telegram and Discord. To run your own bot server, follow these instructions:

• Clone this repository

$ git clone https://github.com/dipu-bd/lightnovel-crawler

• Install calibre for pdf, mobi etc. formats.

  • https://calibre-ebook.com/download

• Install requirements

$ pip3 install --user -r requirements.txt
$ pip3 install --user -r requirements-dev.txt

• Copy .env.example file to .env file. Edit this file and give your API credentials here.

• To run the discord bot:

$ python3 . --bot discord --shard-id 0 --shard-count 1

• To run the telegram bot

$ python3 . --bot telegram

There is a start.sh script to run a bot in ubuntu servers. It will basically execute the python __main__.py and send the task to run in background. I use it to run my discord bot in the server.

$ lncrawl -h
================================================================================
                           📒 Lightnovel Crawler 🍀2.26.0 
                  https://github.com/dipu-bd/lightnovel-crawler 
--------------------------------------------------------------------------------
usage: lncrawl [options...]
       lightnovel-crawler [options...]

optional arguments:
  -h, --help            show this help message and exit

  -v, --version         show program's version number and exit
  -l                    Set log levels. (-l = warn, -ll = info, -lll = debug).
  --list-sources        Display a list of available sources.
  -s URL, --source URL  Profile page url of the novel.
  -q STR, --query STR   Novel query followed by list of source sites.
  -x [REGEX], --sources [REGEX]
                        Filter out the sources to search for novels.
  --login USER PASSWD   User name/email address and password for login.
  --format E [E ...]    Define which formats to output. Default: all.
  --add-source-url      Add source url at the end of each chapter.
  --single              Put everything in a single book.
  --multi               Build separate books by volumes.
  -o PATH, --output PATH
                        Path where the downloads to be stored.
  --filename NAME       Set the output file name
  --filename-only       Skip appending chapter range with file name
  -f, --force           Force replace any existing folder.
  -i, --ignore          Ignore any existing folder (do not replace).
  --all                 Download all chapters.
  --first [COUNT]       Download first few chapters (default: 10).
  --last [COUNT]        Download last few chapters (default: 10).
  --page START STOP.    The start and final chapter urls.
  --range FROM TO.      The start and final chapter indexes.
  --volumes [N [N ...]]
                        The list of volume numbers to download.
  --chapters [URL [URL ...]]
                        A list of specific chapter urls.
  --bot {console,telegram,discord,test}
                        Select a bot. Default: console.
  --shard-id [SHARD_ID]
                        Discord bot shard id (default: 0)
  --shard-count [SHARD_COUNT]
                        Discord bot shard counts (default: 1)
  --suppress            Suppress all input prompts and use defaults.
  --resume [NAME/URL]   Resume download of a novel containing in *****************
  ENV                   [chatbots only] Pass query string at the end of all options. It will be use instead of .env
                        file. Sample: "BOT=discord&DISCORD_TOKEN=***&LOG_LEVEL=DEBUG"

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Open your console and type lncrawl --version first to check if you have installed it properly. Here are some example usage of the app:

• To start an interactive session: lncrawl

• To download using an url: lncrawl -s https://boxnovel.com/novel/reincarnation-of-the-strongest-sword-god/

• To search novels: lncrawl -q "Strongest Sword God"

• To search novels from selected sources: lncrawl -q "Strongest Sword God" --sources

• To download all chapters: lncrawl --all

• To download first 25 chapters: lncrawl --first 25

• To download all between two chapters: lncrawl --range 10 30

• To download all between two chapter links: lncrawl https://boxnovel.com/novel/reincarnation-of-the-strongest-sword-god/chapter-2205 https://boxnovel.com/novel/reincarnation-of-the-strongest-sword-god/chapter-2211

• To download a specific volumes: lncrawl --volumes 2 3

• To define output path: lncrawl -o "D:\Lightnovels\reincarnation-of-the-strongest-sword-god"

• To delete the output folder if exists: lncrawl -f

• To ignore the output folder if exists: lncrawl -i

• To resume download where is has been left previously: lncrawl -i

• To specify output formats: lncrawl --format epub pdf mobi

• To display list of supported sources: lncrawl ---list-sources

• If you provide an option in the argument, it will skip it in the interactive session. If you want to disable all interactive prompts, pass --suppress at the end.

• You can stack up options like this: lncrawl -s https://boxnovel.com/novel/reincarnation-of-the-strongest-sword-god/ -o "D:\Lightnovels\reincarnation-of-the-strongest-sword-god" --last 50 -i --format pdf --suppress

You are very welcome to contribute in this project. You can:

• create new issues pointing out the bugs.
• solve existing issues.
• add your own sources.
• add new output formats.
• create new bots.

• Create new crawler using the sources/_template_.py as template.
• Update Supported sources section in README.md
• Add some test inputs to test_user_inputs variable in lncrawl/bots/test/test_inputs.py

• Create a new bot file using bots/_sample.py as template.
• Import bot to bots/__init__.py file.

Request new one by creating a new issue.

• JSON
• EPUB
• TEXT
• WEB
• DOCX
• MOBI
• PDF
• RTF
• TXT
• AZW3
• FB2
• LIT
• LRF
• OEB
• PDB
• PML
• RB
• SNB
• TCR
• HTML

• Console Bot
• Telegram Bot
• Discord Bot