hurlenko / mloader Goto Github PK
View Code? Open in Web Editor NEWDownload manga from mangaplus.shueisha.co.jp
License: GNU General Public License v3.0
Download manga from mangaplus.shueisha.co.jp
License: GNU General Public License v3.0
In some mangas, the quality of some images is too low compared with others of the same manga. Also the program doesn´t download all the pages of the selected manga. For example, in:
https://manga plus shueisha co jp/viewer/2001777
The program only download 55 pages and it has 58
Hello!
I get this error while using this command: mloader 1002187
11.06.2020 15:30:17 | INFO | mloader.py 164 | Started export
11.06.2020 15:30:18 | ERROR | mloader.py 171 | Failed to save images
Traceback (most recent call last):
File "c:\python38\lib\site-packages\mloader\mloader.py", line 169, in main
loader.save(out_dir, RawExporter if raw else CBZExporter)
File "c:\python38\lib\site-packages\mloader\mloader.py", line 123, in save
exporter: ExporterBase = exporter_cls(dst, title, chapter_name)
File "c:\python38\lib\site-packages\mloader\mloader.py", line 69, in init
self.path.mkdir(parents=True, exist_ok=True)
File "c:\python38\lib\pathlib.py", line 1284, in mkdir
self._accessor.mkdir(self, mode)
NotADirectoryError: [WinError 267] Érvénytelen a könyvtárnév: 'mangaplus_downloads\Samurai 8: The Tale of Hachimaru'
11.06.2020 15:30:18 | INFO | mloader.py 172 | SUCCESS
And this error while using this command: mloader -r 1002187
11.06.2020 15:31:54 | INFO | mloader.py 164 | Started export
11.06.2020 15:31:56 | ERROR | mloader.py 171 | Failed to save images
Traceback (most recent call last):
File "c:\python38\lib\pathlib.py", line 1284, in mkdir
self._accessor.mkdir(self, mode)
OSError: [WinError 123] A fájlnév, a könyvtárnév vagy a kötetcímke szintaxisa nem megfelelő: 'mangaplus_downloads\Samurai 8: The Tale of Hachimaru\#1'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\python38\lib\site-packages\mloader\mloader.py", line 169, in main
loader.save(out_dir, RawExporter if raw else CBZExporter)
File "c:\python38\lib\site-packages\mloader\mloader.py", line 123, in save
exporter: ExporterBase = exporter_cls(dst, title, chapter_name)
File "c:\python38\lib\site-packages\mloader\mloader.py", line 52, in init
self.path.mkdir(parents=True, exist_ok=True)
File "c:\python38\lib\pathlib.py", line 1293, in mkdir
if not exist_ok or not self.is_dir():
File "c:\python38\lib\pathlib.py", line 1403, in is_dir
return S_ISDIR(self.stat().st_mode)
File "c:\python38\lib\pathlib.py", line 1194, in stat
return self._accessor.stat(self)
OSError: [WinError 123] A fájlnév, a könyvtárnév vagy a kötetcímke szintaxisa nem megfelelő: 'mangaplus_downloads\Samurai 8: The Tale of Hachimaru\#1'
11.06.2020 15:31:56 | INFO | mloader.py 172 | SUCCESS
The script had worked perfectly for years, but then entirely stopped on the Monday of last week. The next line after starting, after exactly two seconds, is "Failed to Download Manga".
I think the website might have changed how it hosts images.
Environment:
EDIT: Nevermind, I see this has already been reported.
Or maybe a new repository all together? Maybe call it kmloader?
I don't know what's really the problem here but maybe because it's a new manga so it has had to with maybe the code or something idk, here is the link btw The new manga about sasuke
Stack trace
03.03.2024 16:24:39 | ERROR | main.py 226 | Failed to download manga
Traceback (most recent call last):
File "C:\Users*\AppData\Local\Programs\Python\Python310\lib\site-packages\mloader_main_.py", line 218, in main
loader.download(
File "C:\Users*\AppData\Local\Programs\Python\Python310\lib\site-packages\mloader\loader.py", line 189, in download
self._download(manga_list)
File "C:\Users*\AppData\Local\Programs\Python\Python310\lib\site-packages\mloader\loader.py", line 154, in _download
exporter = self.exporter(
File "C:\Users*\AppData\Local\Programs\Python\Python310\lib\site-packages\mloader\exporter.py", line 140, in init
super().init(*args, **kwargs)
File "C:\Users*\AppData\Local\Programs\Python\Python310\lib\site-packages\mloader\exporter.py", line 47, in init
self._chapter_prefix = self._format_chapter_prefix(
File "C:\Users*\AppData\Local\Programs\Python\Python310\lib\site-packages\mloader\exporter.py", line 70, in _format_chapter_prefix
if Language(language) != Language.eng:
File "C:\Users*\AppData\Local\Programs\Python\Python310\lib\enum.py", line 385, in call
return cls.new(cls, value)
File "C:\Users*\AppData\Local\Programs\Python\Python310\lib\enum.py", line 710, in new
raise ve_exc
ValueError: 7 is not a valid Language
Environment:
mkdir doesn't work since the last update. I have to manually create the mloader_downloads folder to be able to download anything.
Traceback (most recent call last): File "C:\Users\XXXX\AppData\Local\Programs\Python\Python39\lib\pathlib.py", line 1312, in mkdir self._accessor.mkdir(self, mode)
Windows 10
Works fine for normal chapters, but extra (ex) chapter titles are appended to the file name by default. This shouldn't be the expected behavior as --chapter-title
option was not used.
Used command:
$ mloader https://mangaplus.shueisha.co.jp/titles/100041
Part of the filenames:
Summer Time Rendering - c007 (web) [Viz].cbz
Summer Time Rendering - c007x1 (web) [Record 001 Shiori Kobayakawa s Diary Excerpt] [Viz].cbz
Summer Time Rendering - c011 (web) [Viz].cbz
Summer Time Rendering - c011x1 (web) [Record 002 Reviews of Ryuunosuke Nagumo s Swampman] [Viz].cbz
Like the title suggest there are free chapters and mangas which are App exclusive. Maybe you cant implement them.
mloader
works by fetching the chapter data provided by mangaplus api.When there's an extra
chapter, it only denotes itself as extra
, but whose extra
(?), that data is not sent from server. Only the next chapter is given.
And if there are multiple simultaneous extras, the first ones' next chapter value is Always another extra.
So, naming (or more accurately, numbering) the (continuous) extras is hard this way.
# | Original Serial | Currently Downloads As | Expected Behavior |
---|---|---|---|
r2 |
... |
... |
... |
r3 |
Chapter 3 |
c003 |
c003 |
r4 |
extra |
0Exx1 |
c003x1 |
r5 |
extra |
0Exx1 |
c003x2 |
r6 |
extra |
c003x1 |
c003x3 |
r7 |
Chapter 4 |
c004 |
c004 |
r8 |
Chapter 5 |
c005 |
c005 |
r9 |
... |
... |
... |
(or whatever denotation is best for them)
PassedChNum
(just downloaded chapter number, c[0-9]{3}
or d[0-9]{4}
),OldExNum
(number of already downloaded simultaneous extra chapters, [0-9]{1}
, default = 0
)r3
from above table), save that chapter number (as-well-as an extra number) as the above mentioned variables
PassedChNum=c003
OldExNum=0
r4
) is fetched, the previously saved chapter number can be re-used.
OldExNum:
0 > 1
),
c003x1
(using the above 2 variables)PassedChNum=c003
OldExNum=1
r5
) is again an extra
, increase the number in the extra part of the variable,
c003x2
PassedChNum=c003
OldExNum=2
r6
) downloadable chapter
c003x2
PassedChNum=c003
OldExNum=3
r7
) downloadable chapter
c004
OldExNum=0
mloader
🤔Looking forward to see the upcoming changes... 😃
mloader saves all images when using the -raw parameter as PNG when actually (as far as I've downloaded) they're JPG.
This I can verify when opening any image in irfanview and the program will say that it detected that the image is JPG, and it'll ask if I want to rename the extension.
I imagine that it detects this by looking at the file metadata.
Could you make mloader save the images as JPG?
Describe the bug
The LICENSE file specifies GPL3, but the license listed in setup.py
is MIT
Line 36 in 1a64c2e
Expected behavior
Both licenses should match to help packaging mloader.
MIT License (GPLv3)
Stack trace
Environment (please complete the following information):
There are a bunch of non-VIZ translations on M+, so please remove the Viz from filenames.
Title pretty much. Is the project dead?
It seems that now when they put the ID of the chapter it detects it but does not start the download
Is your feature request related to a problem? Please describe.
I would like to crontab mloader to download the last chapter of a list of titles.
Describe the solution you'd like
It would be great to not download a chapter if it has already been downloaded.
Hello,
I would like to download this limited-time free manga, but I encountered an error. I am using pip3. Does this have anything to do with the pip version? Here is the error message.
mloader --chapter-subdir --chapter-title -o my_manga -c 10051 https://ynjn.jp/viewer/92
_ _
_ __ ___ | | ___ __ _ __| | ___ _ __
| '_ ` _ \| |/ _ \ / _` |/ _` |/ _ \ '__|
| | | | | | | (_) | (_| | (_| | __/ |
|_| |_| |_|_|\___/ \__,_|\__,_|\___|_|
25.01.2024 17:12:05 | INFO | __main__.py 206 | Started export
25.01.2024 17:12:06 | INFO | loader.py 138 | 1/1) Manga:
25.01.2024 17:12:06 | INFO | loader.py 139 | Author:
25.01.2024 17:12:06 | ERROR | __main__.py 226 | Failed to download manga
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/mloader/__main__.py", line 218, in main
loader.download(
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/mloader/loader.py", line 189, in download
self._download(manga_list)
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/mloader/loader.py", line 144, in _download
chapter = viewer.pages[-1].last_page.current_chapter
~~~~~~~~~~~~^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/protobuf/internal/containers.py", line 93, in __getitem__
return self._values[key]
~~~~~~~~~~~~^^^^^
IndexError: list index out of range
25.01.2024 17:12:06 | INFO | __main__.py 227 | SUCCESS
manga url is : https://ynjn.jp/viewer/92/10051
From #36 (comment) :
I'm also curious - if I were to pay for subscription, would it be recognized by the script and let me download the completed series? Does secret key carry that information?
please help
Describe the bug
The PyPI tarball doesn't include the requirements.txt
which is required by setup.py
Line 19 in 1a64c2e
$ pip install mloader
...
Traceback (most recent call last):
File "/private/tmp/nix-build-mloader-1.1.8.drv-0/mloader-1.1.8/nix_run_setup", line 8, in <module>
exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
File "setup.py", line 19, in <module>
with open("requirements.txt", "r", "utf-8") as f:
File "/nix/store/76b5vjb1wkl7b38g8pazwv0p6pxmi53f-python3-3.10.5/lib/python3.10/codecs.py", line 905, in open
file = builtins.open(filename, mode, buffering)
FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'
Expected behavior
Be able to install mloader from PyPI
Stack trace
Environment (please complete the following information):
This issue shall be laid out in three parts in order of ease of fixing, with the third part being more of an enhancement / suggestions. It can also serve as a good template for a prospective pull request to reference against.
The format_image_name
function currently erroneously formats files with two .
characters, due to preceding {ext}
with .
while defaulting / expecting ext
to include the .
of the extension.
Fix: Remove the .
in the format string or set ext
to be ext.lstrip('.')
to allow for ext = ".png"
and ext = "png"
to both be valid.
It would be highly preferable for the created cbz archive to be of a folder containing the individual pages rather than of the files themselves. I.e. rather than the layout of manga-chapter.cbz
being manga-001.png
, manga-002.png
,... it should be manga-chapter/manga-001.png
.
Fix: Change add_image
to use str(Path(self.path.stem)/self.format_image_name(index)))
for the first argument to writestr
. Alternatively, str(self.name.relative_to(self.name.parent).with_suffix('') / self.format_image_name(index))
would work without having to recast stem
to Path
. Either way it looks ugly for a one liner.
This one is the biggest set of changes though mainly deals with the same two functions from above. Currently none of the filenames used by mloader conform to current best practices. Chief among them is that the individual pages do not include their chapter number in their names. Combined with the current issue that the archives are flat (above) this is just asking for files to get clobbered by accident. For best standards refer to Daiz.
This would mainly require changing format_image_name
and the initial cbz creation line in CBZExporter
to include info about chapter number and some extra formatting. The most stringent conforming implementation would have the following layout for chapter 29 of a manga named "Foo" consisting of png pages (for simplicity).
Foo - c029 (web) [Viz]/
Foo - c029 (web) [Viz]/Foo - c029 (web) - p001 [Viz].png
...
Foo - c029 (web) [Viz]/Foo - c029 (web) - p022 [Viz].png
(If the chapter number exceeds 999 the numbering becomes d1000
starting at the 1000th chapter)
Fix: It is, of course, personal choice if you wish to fully conform to existing best practices. However, at the very least it would be helpful to modify format_image_name
to take the chapter number as well (or make a version specifically for CBZExporter
which already has said info) and format pages as {self.title} - c{self.chapnum:0>3} - p{self.index:0>3}.{ext}
(assuming the extension issue is fixed by specifying ext
as "png"
instead of ".png"
).
hello I have followed you're project and I'm trying to make a simple script to decode one image
unfortunately it doesn't work I get the image with the same size but it's not readable
I think I missed something if you can give me some help please, thank you
url = 'https://mangaplus.shueisha.co.jp/drm/title/100170/chapter/1009902/manga_page/super_high/7052269.jpg?key=e3749f6bba63f862625065a35551bf54&duration=86400'
session = requests.session()
session.headers.update(
{
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; "
"rv:72.0) Gecko/20100101 Firefox/72.0"
}
)
resp = session.get(url)
data = bytearray(resp.content)
key = bytes.fromhex('e3749f6bba63f862625065a35551bf54')
a = len(key)
for s in range(len(data)):
data[s] ^= key[s % a]
path = Path('')
filename = Path('test.jpg')
path.joinpath(filename).write_bytes(data)
Describe the bug
When i try to type this exact command mloader -l -o "D:\manga" -t 500001
, the script gives me this error
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "C:\Users\dougl\scoop\apps\python\current\Scripts\mloader.exe\__main__.py", line 4, in <module>
File "C:\Users\dougl\scoop\apps\python\current\Lib\site-packages\mloader\__main__.py", line 10, in <module>
from mloader.exporter import RawExporter, CBZExporter
File "C:\Users\dougl\scoop\apps\python\current\Lib\site-packages\mloader\exporter.py", line 8, in <module>
from mloader.response_pb2 import Title, Chapter
File "C:\Users\dougl\scoop\apps\python\current\Lib\site-packages\mloader\response_pb2.py", line 34, in <module>
_descriptor.FieldDescriptor(
File "C:\Users\dougl\scoop\apps\python\current\Lib\site-packages\google\protobuf\descriptor.py", line 560, in __new__
_message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
1. Downgrade the protobuf package to 3.20.x or lower.
2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
Expected behavior
I expected the script to download the manga normally, like it did before
Environment:
Describe the bug
Doing any command in mloader including --help
results in the error displayed on the stack trace.
Expected behavior
Expected to see help information
Stack trace
╰─ mloader --help
Traceback (most recent call last):
File "/home/subject/.local/bin/mloader", line 5, in
from mloader.main import main
File "/home/subject/.local/lib/python3.8/site-packages/mloader/main.py", line 10, in
from mloader.exporter import RawExporter, CBZExporter
File "/home/subject/.local/lib/python3.8/site-packages/mloader/exporter.py", line 8, in
from mloader.response_pb2 import Title, Chapter
File "/home/subject/.local/lib/python3.8/site-packages/mloader/response_pb2.py", line 21, in
create_key=_descriptor._internal_create_key,
AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'
Environment (please complete the following information):
M+ added french releases recently to their site, is it possible to add support for french language please ?
Change in viz url format leads to error/failure. No longer using the */viewer/# format format chapters for example. Now uses */title-chapter-#/chapter/#?action=read format.
Steps to replicate
Install the develop
branch, change AUTH_PARAMS
, then run the following:
$ mloader --format raw --chapter 1017028
Sakamoto Days - 000 (web) [Oneshot] [Unknown]
This chapter's name is "One shot". This is probably the cause.
Environment:
As part of my workflow I take the downloaded chapters and insert a ComicInfo.xml into the cbz files. However, right now this step has to be manually generated and requires a bunch of work for things like dates a chapter was released. It would be awesome if some or all of that information could be generated since it should all be available on the site.
Something like adding a "--metadata" flag, to create a file with author, and date for each chapter.
When downloading a manga through the app, if there are any punctuation (like :
or '
etc. ), in the Manga's Title or Chapter's Title, they are just replaced with a white-space.
It would be good if the formatting for naming images/cbz/folder should've retained some of the punctuation to make it look more readable.
https://mangaplus.shueisha.co.jp/titles/100248
Manga Name: Nue's Exorcist
, but it results as Nue S Exorcist
For Chapter's names: :
are deleted and '
are replaced by a white-space
'
(So that the manga title should be Nues Exorcist
at least):
with -
along with previous fix
Chapter 23: Nue's Candidate
is saved with Chapter 23 Nue s Candidate
, it should be Chapter 23 - Nues Candidate
If these can be implemented, please do so 🙏🏼
Hello, just wondering was there a particular reason you wouldn't do this? I know the sub-title comes in various prefix like [Number #n:..., Chapter #n:..., #n:]. But its still possible to filter and omit all of those with regex and just grab the following sub-title, right?
Would be good as a reference for jumping between chapters.
Great program, but it always download the full chapters, even though you already put chapter id.
chapter id = number in chapter url
Did I get it wrong?
Any way to hide the display of the files you have already downloaded on the screen and only display the process of the new one being downloaded, so as not to always fill the entire screen with the entire downloaded list?
I'm trying to download some currently hidden chapters for title:200020 at once, is there a way to do this?
mloader saves images with a trailing dot which causes some issues with image viewer programs I'm using.
Hi,
I was recently writing an arch PKGBUILD for mloader and ran into some issues with the requirements.txt
. This is a minor enough change that rather than opening a PR I felt an issue made more sense.
Issue:
Currently mloader's requirements.txt
restricts to exact package versions with ==
. This is most likely not intentional, seeing as how it seems to run fine with newer versions of the required packages. In practice this means on systems with newer versions of the required packages any install from python setup.py install
will fail at run time (I assume pip to just install multiple package versions).
Proposal:
Change requirements.txt
to use >=
or ~=
instead of ==
. I.e. sed -i.bak 's/==/>=/' requirements.txt
(or ~=
instead of >=
, should be equivalent here)
P.S. Let me know if you are interested in the PKGBUILD / have any preferences about mloader's being in the Arch User Repository (AUR) or not.
Describe the bug
As stated in the title, it fails to get chapter ids when trying to download an entire title.
Arguments:
-t 100235 -r
Output:
03.04.2023 01:49:40 | INFO | __main__.py 206 | Started export
03.04.2023 01:49:42 | ERROR | __main__.py 223 | Failed to download manga
Traceback (most recent call last):
File "C:\Users\{redacted}\mloader\__main__.py", line 215, in main
loader.download(
File "C:\Python311\Lib\site-packages\mloader\loader.py", line 191, in download
manga_list = self._normalize_ids(
^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\site-packages\mloader\loader.py", line 115, in _normalize_ids
details.chapter_list_group.first_chapter_list,
^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'TitleDetailView' object has no attribute 'chapter_list_group'
03.04.2023 01:49:42 | INFO | __main__.py 224 | SUCCESS
Environment:
From #36 (comment):
As you probably know Shueisha locked some series behind a paywall of subscription. These series only have 3 first chapters available and after downloading them, the script throws an error. Assassination Clasroom (100001) is an example. Would be great if the error could throw some human readable message.
Hello,
Could you share how you created the response protobuf file? I want to create a similar application for a different site but I don't have a lot experience scrapping sites.
Also, how did you figure out the decryption method?
Thanks
I don't know if they have updated the web or what exactly it is because until yesterday the downloads were working without problems.
Here is what the mloader tells me when downloading
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.