metafates / mangal Goto Github PK
View Code? Open in Web Editor NEWπ The most advanced (yet simple) cli manga downloader in the entire universe! Lua scrapers, export formats, anilist integration, fancy TUI and more!
License: MIT License
π The most advanced (yet simple) cli manga downloader in the entire universe! Lua scrapers, export formats, anilist integration, fancy TUI and more!
License: MIT License
Since Mangal tends to be cross-platform, it would be a nice addition to add support for FreeBSD systems
FreeBSD support + prebuilt binaries with package manager install option (?)
Try to build from source using go
It may be compatible already (if building from source) but I don't have access to any freebsd machine right now (nor time to setup vm)
Describe the bug
looking for mangas and reading chapters isn't working properly
To Reproduce
Steps to reproduce the behavior:
Chapters\n no items
Expected behavior
Should work
System
Additional context
This is using the default provider (manganelo) and both pdf and cbz as filetypes
Mangal panics when using mangal --config ...
panic: runtime error: invalid memory address or nil pointer dereference
Searching for manga always returns no results, even though the device is connected to the Internet.
Running mangal doctor
yields a very weird result as well:
(I have the latest version, installed with curl -L -o mangal https://github.com/metafates/mangal/releases/download/v2.2.0/mangal-android-arm64
curl -L -o mangal https://github.com/metafates/mangal/releases/download/v2.2.0/mangal-android-arm64
chmod u+x mangal
to make execution possible../mangal config init
./mangal inline -q "berserk"
returns no results as well as any other search queryMangal should display a list of mangas to read
I'm running Termux v0.118.0 from F-droid on a OnePlus 7 Pro running Android 11
Edit: The same thing happens on the version that can be downloaded from Github
Linux
mangal doctor
and it didn't helpEdit: fixed some typos
i get this error running on arch linux
tried both installing through brew and using the binary
"https://m.manganelo.tv/search/berserk": dial tcp: lookup
m.manganelo.tv on [::1]:53: read udp [::1]:42958->[::1]:53: read:
connection refused
stack traceback:
[G]: in function 'error'
/home/luc/.config/mangal/sources/manganelo.lua:75: in
function 'AssertFalse'
/home/luc/.config/mangal/sources/manganelo.lua:11: in
main chunk
[G]: ?
while trying to run any search, even custom scrapers
Search among multiple sources at the same time
Select what sources to use for search in the TUI by pressing select key
No response
Originally posted by @omaru-ok in #7 (comment)
Is your feature request related to a problem? Please describe.
Describe the solution you'd like
Describe alternatives you've considered
Additional context
Is your feature request related to a problem? Please describe.
ComicInfo.xml is a metadata file used by some comic readers. It holds information about comic such as series name, chapters count, reading direction and so on. You can read more about it here
Describe the solution you'd like
An option in config file that would specify whether to create ComicInfo.xml file or not
Describe alternatives you've considered
None
Additional context
Even though this format is very rarely used still this would be a nice addition to existing functionality
After a quick apt-get install golang-go i get those errors on installation:
go install github.com/metafates/mangal@latest
can't load package: package github.com/metafates/mangal@latest: can only use path@version syntax with 'go get'
go get github.com/metafates/mangal@latest
go get github.com/metafates/mangal@latest: can't request explicit version of path in main module
Infos:
Go Version: go1.13.8 linux/amd64
System: Ubuntu 20.04.4 LTS on Windows 10 x86
Mangakakalot (built-in provider) shows no chapters.
To see chapters
If you try to restart mangal after unsuccessful search and open the same manga you will see chapters. Weird.
Use manganelo for now, it's has the same mangas as mangakakalot
Uncompressed sources along with fast cache
Can you please add the usage of their api
Since both of these sources have the most chapters
No response
No response
Is your feature request related to a problem? Please describe.
when I finish my reading session using the read option, I need to either manually go to anilist's website and update my reading progress there or use an external custom script which makes a post request to anilist's api with my tokens
Describe the solution you'd like
if possible, i would like a solution integrated withing this project itself
Describe alternatives you've considered
Anilist's website manually or external custom script
I created an AUR package for mangal from the release binaries: https://aur.archlinux.org/packages/mangal-bin
Please add a link to this package in the README under Installation > Linux.
When downloading files on windows it nevers creates the file with a file extension (e.g. .pdf)
Even when specify file type via config or CLI it just saves them as a file (no extension)
Having All chapters inside a single volume
pack all chapter inside a single volume cbz {with option if possibel to add a ongoing volume {because if user adds a volume that is ongoing, adds again , the client would have to unzip , check existing , and add new images of the latest chapter , further complicating the process, unless dev can possibly give a try}
extracting images out of the cbz in the volume folder and repacking furhter outsourcing a internal feature.
No response
Is your feature request related to a problem? Please describe.
It would be very convenient to be able to check if new version was released without going to GitHub page manually
Describe the solution you'd like
A command like mangal check-update
Describe alternatives you've considered
Use package manager (scoop, Homebrew)
Smol chapters without able to add meta data from comictagger, even thou komf exist to add data from other places into komga
If possible packing chapters into volumes
Making a custom script to do that, but a built in support for some sources would be nice, can probably be used with managdex api to do the same with other sources to fetch chapters per volume
No response
Running the completion command (mangal completion
) tries to create the main configuration file and fails if not possible.
This becomes an issue when we want to package mangal in an isolated build environment, without a user home, as mangal fails with a panic: mkdir /homeless-shelter: permission denied
error.
You can reproduce the same behavior restricting the ~/.config/mangal
folder permissions when using mangal, i,e, setting the configuration folder owner to root.
Mangal will then try to create a folder in /homeless-shelter
.
The completion command should be available without requiring the generation a configuration file.
I'm currently packaging it for Nix. You can also reproduce the same behavior in the branch mangal-permission-error
of this repository using nix build .#mangal
using a nix flakes or nix-build -A mangal
from either a Linux distribution or MacOS with nix installed.
As a workaround I'm setting the HOME
variable to a temporary folder while building the app.
Linux
When trying to read chapter from history with mini mode mangal will panic
mangal mini --continue
The same as mangal --continue
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x20 pc=0x102fbed80]
goroutine 54 [running]:
github.com/metafates/mangal/downloader.Read(0x14000879ec0, 0x14000171f88)
github.com/metafates/mangal/downloader/read.go:24 +0x60
github.com/metafates/mangal/mini.(*mini).handleChapterReadState.func1(0x14000879ec0, 0x14000482cc0, 0x0, 0x1)
github.com/metafates/mangal/mini/states.go:240 +0x78
created by github.com/metafates/mangal/mini.(*mini).handleChapterReadState
github.com/metafates/mangal/mini/states.go:302 +0x278
macOS
Sometimes there can be more than one anilist match and if we are using the inline mode, it is not possible to select one.
Can you add an ability to pass an anilist id in inline mode?
No response
Describe the bug
When trying to read a chapter that has a colon in title nothing happens (e.g. "Chapter 1 : Lorem Ipsum")
To Reproduce
Steps to reproduce the behavior:
Expected behavior
It is expected to be opened in the default reader app
System
Additional context
When downloading such chapters everything is fine, this happens only when trying to use the read option
When I press r
on a manga to read the single chapter, Mangal downloads it to a temporary directory on my Mac.
I think it would be better for the manga downloaded with this option to be saved in the directory specified in the config file.
Another plausible solution would be to auto-delete the temporary file when quitting Mangal.
Mangasee
scraper (with mangal install
)mangal inline -S Mangasee -q "death note" -m first -c 0 -p -j
In this case it should be .png
Mangasee scraper has nothing to do with it, it's just an example
macOS
Download chapter get "Image: Unknown format" error
Download complete
No response
Linux
Some custom scrapers fail to download pages due to exceeding http timeout of 10 seconds, even though pages are downloading normally (just slow)
No response
To have 20-30s of timeout
Greetings, glorious mangal
maintainers! I quietly package awesome software for Gentoo Linux. Because mangal
is clearly both awesome and software, packaging mangal
for Gentoo is what I am now doing.
Except... I can't. Cue sadness cat.
The mangal
repository (and thus source tarballs derived from this repository) currently omits the top-level vendor/
directory required to reproducibly build and install mangal
from source.
That's usually fine. In the absence of a vendor/
directory, go
subcommands silently download and cache dependencies listed in go.mod
into the user-local module cache (e.g., ~/go/pkg/mod/
). Go developers and brave end users manually building mangal
from scratch are accustomed to that.
That's less fine for foolhardy packagers (such as myself) attempting to package mangal
for source-based redistribution at the system level. But "source-based redistribution at the system level" is the literal definition of Gentoo Linux. In other words, Gentoo isn't happy. Let's make Gentoo happy.
Most system-level package managers (e.g., emerge
under Gentoo Linux) and language-level package managers (e.g., the Swift Package Manager (SPM)) sandbox at least filesystem access for security. Increasingly many also sandbox network access for the same reason. This is where our packaging trouble starts and ends.
Since mangal
omits a vendor/
directory, go
subcommands silently:
Admittedly, Gentoo can trivially circumvent filesystem sandboxing by just (A) caching to a temporary non-sandboxed work directory while building and then (A) deleting that directory after building. That's fine.
Gentoo cannot trivially circumvent network sandboxing, however. In theory, Gentoo could vendor mangal
dependencies into a new source tarball that we host and maintain on your behalf... which is the problem. Gentoo ...okay, I mean me doesn't want to host and maintain separate source vendor tarballs for every single Go application in existence.
We're lazy and slovenly. Vendoring Go dependencies is non-trivial, error-prone, time-consuming, and anti-fun. Moreover, every single other Linux distribution (of which there are a countably infinite number) would need to reduplicate that same work. This is probably why most Linux distributions that currently package mangal
sidestep this issue by just redistributing your binary Linux executables. That is what Arch does, for example. But that's bad β or at least fundamentally incompatible with the Gentoo religion.
We're zealous fanatics about source-based installation. We couldn't be wrong! π
Omitting vendor/
is typically the best practice for Go middleware like libraries and frameworks. Conversely, committing vendor/
is typically the best practice for actual Go applications shipping executable binaries... like mangal
. Sources or it didn't happen, so:
The answer to the age-old question of, βshould I commit the vendor dependencies in my Go project to source control?β is βalmost always.β
For my own peace of mind I would always prefer to commit the
vendor/
directory alongsidego.mod
andgo.sum
...
Very well. I can see that neither the lolcat nor the two blog articles I linked to above have convinced you. In that case, I shall now bludgeon you with verbose words.
There's really only one valid reasons to omit vendor/
: repository size. But that's not a particularly valid reason in the modern era, because GitHub ...so, Microsoft is what I'm saying is already committed to taking the bandwidth and storage hit on your behalf.
There are many valid reasons to commit vendor/
, however β including:
mangal
builds are currently non-deterministic. By deferring to go
subcommands that bang on remote servers for online installation, mangal
builds tacitly assume that URLs, versions, and licenses are perfectly stable across all dependencies β forever. But that's never the case. Packaging is a constantly shifting landscape wrapped in a badly documented enigma inside a vanishing mystery of 404
errors. The Node.js world discovered this the rough way when left-pad
spontaneously blew everything up.go
no longer needs to bang on remote servers for online installation. Since requisite dependencies are locally available under vendor/
, CI/D turnaround (and thus turnaround for tests and PRs) measurably improves.Thanks so much for all the automated manga downloading, mangal
magicians. My manga backlog may currently be crying, but everyone here is a spectacular gift to humanity.
Thanks for making a great tool!
When downloading with inline, for example:
inline --source "Sourcename" --query "Manganame" --manga first --chapters all -d
It seems like all chapters get downloaded every time, which isn't great if you are using mangal to do batch downloads of multiple series.
Would be awesome if there is a --check-local flag or something like that.
All that's really needed is to check if the file exists before downloading.
I guess there could be instances where a chapter is updated, which then would not be downloaded.
That would not an issue for me at least.
No response
From what I can see, this is not a feature yet.
Hopefully I'm wrong, and in that case I apologize for creating this issue.
Describe the bug
Chapters are failing to download in cbz format if plain folder exists
To Reproduce
Steps to reproduce the behavior:
mangal -f plain
)mangal -f cbz
)Expected behavior
It is expected to not produce any errors and just download chapters
Screenshots
System
Additional context
This error appears only for this combination and sequence of formats. (e.g. cbz -> plain works fine, but plain -> cbz fails)
Hi, The Docker file i linked was just that an example to build based on, not meant to be used publicly as it might lead to permissions issues.
FROM alpine:latest
ENV VERSION=v2.0.0
ENV USER=abc
ENV UID=1000
ENV GID=1000
ENV XDG_CONFIG_HOME=/config
WORKDIR /config
VOLUME /config
RUN addgroup -g "$GID" "$USER"
RUN adduser \
--disabled-password \
--gecos "" \
--home "$(pwd)" \
--ingroup "$USER" \
--no-create-home \
--uid "$UID" \
"$USER"
ADD https://github.com/metafates/mangal/releases/download/${VERSION}/mangal-linux-amd64 /usr/local/bin/mangal
RUN chmod +x /usr/local/bin/mangal
USER abc
ENTRYPOINT ["/usr/local/bin/mangal"]
This version is more appropriate for public as it run the tool in non-root user context.
No response
Please do take look at this build file i use for my own tool,
https://github.com/ArabCoders/watchstate/blob/master/.github/workflows/build.yml
it will help you make an auto build.
It will help greatly to also add some environment variables for data locations i.e. MANGAL_PATH
for config storage, MANGAL_DOWNLOADS
for where data is stored.
When I run mangal I would like to continue reading where I left off during my last session
A -c | --continue flag where I can select a manga and which will automatically take me to where I last left off
No response
No response
It would be nice to have an option to integrate Mangal into scripts.
Something like that
mangal inline --source "..." --search "..." --manga 1 --from 1 --to 42 --download
Use mangal mini
and do some magic with stdin. But idk if it's even possible
No response
Right now it seems the tool uses whatever the chapter name is as filename, which is good idea, however some manga and or chapters uses characters which are illegals in windows.
For example, Lets take Berserk as example the first 3 chapters are named:
'[1] Vol.1 Chapter 0.01: The Black Swordsman.pdf'
'[2] Vol.1 Chapter 0.02: The Brand.pdf'
'[3] Vol.1 Chapter 0.03: The Guardians Of Desire (1).pdf'
they works correctly in Linux host, however viewing them via windows produce file names like
you could see list of those characters here for example
No response
No response
No response
Windows
It would be awesome to see what chapters were read on Anilist inside Mangal.
Some mark next to the chapter title that would indicate that this chapter was read.
Also, an option in config to enable/disable this behavior
No response
No response
When using invalid format from --format flag mangal won't warn you about it and will crash later
mangal --format something
It is expected to see that message
When using invalid format from the config file everything works as expected. This issue is only related to the command line argument
macOS
mangal doctor
and it didn't helpIs it possible to add a parameter for searching manga in inline mode?
No response
No response
No response
Describe the bug
When pressing select button (space or enter) on empty list (e.g. when nothing was found) mangal crashes
To Reproduce
Steps to reproduce the behavior:
Expected behavior
It is expected that nothing happens
System
Is your feature request related to a problem? Please describe.
I would like to read manga in ebook readers and having an option to export manga as .epub
and other formats would be very helpful
Describe the solution you'd like
An option for format
in config
Describe alternatives you've considered
Download as pdf or plain images and then use an external script or online converter
I'm always frustrated when there are 0 build instructions.
I want to build this app on an ARM device, but can't find any instructions.
I am not a Go developer, but I can use Dockerfiles. Usually, they build the binaries from source. Otherwise, what's the point? I can download the binary myself, no need for a Docker image.
Shell instructions that build the binary would be nice. An official Dockerfile that runs these instructions would be even better.
Both would be perfect.
I see a Makefile, but don't really know how to use these. Do I build the binary by executing make
?
Typically, the basics of a Dockerfile are to put shell instructions into lines e.g.
RUN mkdir /aaa
RUN echo 123
and to execute the binary in the end with CMD
or ENTRYPOINT
or both.
Current scraping system is very weak and unstable with a lot of restrictions. If one site gets blocked it is very complicated to find a new one that would pass all the requirements. So I propose to use embedded scripts that would allow to define more complex actions.
Ferret is a declarative query language. It has the ability to scrape JS rendered pages, handle all page events and emulate user interactions.
Syntax looks like that
LET doc = DOCUMENT('https://github.com/topics')
FOR el IN ELEMENTS(doc, '.py-4.border-bottom')
LIMIT 10
LET url = ELEMENT(el, 'a')
LET name = ELEMENT(el, '.f3')
LET description = ELEMENT(el, '.f5')
RETURN {
name: TRIM(name.innerText),
description: TRIM(description.innerText),
url: 'https://github.com' + url.attributes.href
}
Integrate Lua scripts with Gopher Lua. But that is way more complicated than Ferret and unnecessary to be honest
Anko is a great alternative!
No response
I'm trying to download manga to a specific folder, but mangal keeps downloading to some temporary folder instead regardless of what I put in the download path. I am using the default config file (from config init) with only the download path changed to
download_path = '/mangal/someFolder'
Chapter should be downloaded to the specified filepath.
No response
macOS
mangal doctor
and it didn't helpDescription
When downloading any manga via Powershell or Command Line, only blank files with 0B get downloaded, yet it states that the download was successfull. Also if you download via powershell you can't remove them via explorer.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Standard file type gets downloaded to default location (PDF file to home folder).
Screenshots
Image
Desktop:
Hi, I've made an updated docker container build to take advantage of the new ENV variables.
FROM alpine:latest
ENV MANGAL_VERSION=v2.1.1
ENV MANGAL_CONFIG_PATH=/config/mangal.toml
ENV MANGAL_DOWNLOAD_PATH=/downloads
ENV MANGAL_USER=abc
ENV MANGAL_UID=1000
ENV MANGAL_GID=1000
WORKDIR "/config"
RUN mkdir -p "${MANGAL_DOWNLOAD_PATH}" && addgroup -g "${MANGAL_GID}" "${MANGAL_USER}" && adduser \
--disabled-password \
--gecos "" \
--home "$(pwd)" \
--ingroup "${MANGAL_USER}" \
--no-create-home \
--uid "${MANGAL_UID}" \
"${MANGAL_USER}" && \
chown abc:abc /config "${MANGAL_DOWNLOAD_PATH}"
ADD https://github.com/metafates/mangal/releases/download/${MANGAL_VERSION}/mangal-linux-amd64 /usr/local/bin/mangal
RUN chmod +x /usr/local/bin/mangal
USER "${MANGAL_USER}"
ENTRYPOINT ["/usr/local/bin/mangal"]
I think MANGAL_CONFIG_PATH
is kind of incorrectly named, as of v2.1.1
it points to a FILE
directly not a PATH
, this might cause misinterpretation as i did when i was updating my docker container. I think it should take PATH
which gives you the possibility in the future to migrate config files when breaking changes happens.
For example, right now config is at mangal_v1.toml
, in the future you make breaking change, you can create mangal_v2.toml
and migrate settings from mangal_v1.toml
.
Also, making more config options configurable via ENV variables is really helpful. prime example at least in my case is formats.default
i always set it to cbz
If the date field is incomplete in anilist (for example year and month exist but the day is null), mangal writes 0
as the value in ComicInfo.xml. This causes komga to fail.
You can see an example manga in this link
It should not put an invalid value for dates.
No response
Linux
It would be awesome to have a powershell script to run mangal just like this shell script does
Here's my attempt to make it run.ps1 but I was not able to test it (I don't have a windows machine and setting up windows vm on m1 is a pain) and I'm not familiar with PowerShell.
If someone wishes to test it or point out some issues with it I would be very grateful
Title says its all
[downloader]
path = "$XDG_CONFIG_HOME/..."
# or any other variable like $MANGA_PATH
Use env-based configuration rather than file one
No response
I added custom scrapper for my source but nothing got scrapped, or yes. I really don't know what is happening or any information to fix that problem. All I got was
Checking if latest version is used... OK
Checking config... OK
Please, enter a manga title to test customsource: isekai
Checking source customsource... Fail
Source customsource is not available
Try to reinitialize your config with mangal config init --force
Note, that this will overwrite your current config
Why did the source not available? How can the default source solve my problem?
Add a debug flag or atleast write the log somewhere
No response
No response
I need to know where logs (example: search or find selector) are located
thanks :)
Is your feature request related to a problem? Please describe.
It would be cool to be able to grab entire volumes instead of by chapter.
Describe the solution you'd like
A flag or something to add a step to roll up chapters into a single pdf for each volume.
Describe alternatives you've considered
n/a
Additional context
Obviously pretty dependent on structure of the manga since some don't use volumes but just an idea..
When using a range selector --chapters x-y
it will select all available chapters
mangal inline --source Manganelo --query "death note" --manga first --chapters 1-3 -d
Select only specified chapters
No response
macOS
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.