mgmcn / pppixiv Goto Github PK

View Code? Open in Web Editor NEW

137.0 137.0 5.0 6.35 MB

A tool to download pixiv illustrations

License: Apache License 2.0

Dockerfile 1.27% Python 54.30% Shell 0.40% CSS 8.95% JavaScript 22.93% HTML 12.14%

pixiv-downloader

pppixiv's Introduction

🤪 Hi there 👋

😄 I’m currently working on P2P system.
😍 I am an enthusiast of web crawling technology.
🤩 I enjoy contributing to open source libraries.

🔨 Programming languages used

Most of the programming languages were learned when I was an undergraduate. Some of these programming languages I use regularly and some I rarely use anymore.

Level	Languages
Frequently used
Occasionally used
Rarely used

🔧 Other technologies used

Type	Technologies
Framework
Database
Middleware
Container technology
Test
CI/CD
Cloud service
Embedded device

🔥 Contributions to the open source community

Repository	Stars	Forks	Contributions
go-libp2p			#1661pr
ProxyPool			#186pr、#187pr
get-pixivpy-token			#64pr、#67pr
advanced-go-programming-book			#635pr
cmd-wrapped			#22pr
hugo-PaperMod			#1416pr

👥 Co-developed

Repository	Stars	Forks
pppixiv
tg-dumb-bot

🧸 Useless toys

Repository	Summary
bilibiliWordCloud	Self-use tool for generating animation short comment keywords
deleteHelper	Self-use deleteHelper
FastCV	Help beginners build their personal cv with github.io
fastDownloader	Self-use crawler downloader
hugoThemesRanking	Self-use tool to quickly retrieve hugo themes ranked by stars
mazebot	Q-learning demo
P2P-File-Sharing	LAN File Sharing Tool

⚙️ GitHub Analytics

pppixiv's People

Contributors

Stargazers

Watchers

Forkers

rio98 xc1996114514 yoho321 liulime weiqizhang1999

pppixiv's Issues

About download link

Note that the download link you get from our getIllustDownloadUrl api must be downloaded by our program's pixivAPI.
Actually ,this pppixiv project is only a sub-project of the tg-bot project.
But we will continue to develop new features for this project.
Plz stay tuned!

Adding a test case of a failed request

Our current test.py will only request id and mode that exist.
So what if someone posts an id and mode that doesn't exist?

Set the token to be refreshed every hour

The first time we log in to pixiv, we are given a token that expires in 3600 seconds, and we need to refresh it before it expires, which means we don't have to wait until it expires to log in to pixiv again to get the token, which is slow, but refreshing it is fast.

Using keyword(tag) searching illustrations

For example👇

json_result = self.pixivApi.search_illust('eva', search_target='partial_match_for_tags')

A bug may occur when the file name is duplicated

A bug may occur when multiple requests download a file with the same name at the same time.(although the file contents are different)
Solution, add a random number to the file name!
But not graceful！

Random antonym & Zoo girl tags

When requesting illustrations with a specific tag, there is a 20% chance that it will return illustrations with antonym tags
Add a shortcut for requesting tags from "Zoo girls", i.e., "{} girl".format(np.random.choice([cat, fox, wolf, ...]))

Retrieve trending tag/top-ranked illustrations

As mentioned in the title.

Remove some unnecessary download options

see -> requirements.txt

For users blocked by GFW

Proxy settings required!

Need someone to help us improve the front-end, because we are all new to front-end.

As title mentioned.

Need someone to test gfw mode in mainland china

Remove unnecessary downloaded libs. (requirements.txt file)

Flask version bug

As title mentioned!

gppt needs to be fixed

2023-09-02 13:38:54 [2023-09-02 04:38:54,754] DEBUG in main: =====================================================================================
2023-09-02 13:38:54 [2023-09-02 04:38:54,754] DEBUG in main: Authorized on pixiv account [email protected]. Please wait for authentication.
2023-09-02 13:39:15 Traceback (most recent call last):
2023-09-02 13:39:15   File "/APP/main.py", line 26, in <module>
2023-09-02 13:39:15     success, msg = app.run_services()
2023-09-02 13:39:15                    ^^^^^^^^^^^^^^^^^^
2023-09-02 13:39:15   File "/APP/app.py", line 21, in run_services
2023-09-02 13:39:15     success, msg = self.myPixiv.start_pixiv_session()
2023-09-02 13:39:15                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-09-02 13:39:15   File "/APP/services/pixiv.py", line 225, in start_pixiv_session
2023-09-02 13:39:15     success = self.get_token()
2023-09-02 13:39:15               ^^^^^^^^^^^^^^^^
2023-09-02 13:39:15   File "/APP/services/pixiv.py", line 54, in get_token
2023-09-02 13:39:15     res = self.pixivTokenApi.login(headless=True, username=self.pixivUsername, password=self.pixivPassword)
2023-09-02 13:39:15           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-09-02 13:39:15   File "/usr/lib/python3.11/site-packages/gppt/_selenium.py", line 93, in login
2023-09-02 13:39:15     self.__try_login()
2023-09-02 13:39:15   File "/usr/lib/python3.11/site-packages/gppt/_selenium.py", line 163, in __try_login
2023-09-02 13:39:15     el = self.driver.find_element(
2023-09-02 13:39:15          ^^^^^^^^^^^^^^^^^^^^^^^^^
2023-09-02 13:39:15   File "/usr/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 830, in find_element
2023-09-02 13:39:15     return self.execute(Command.FIND_ELEMENT, {"using": by, "value": value})["value"]
2023-09-02 13:39:15            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-09-02 13:39:15   File "/usr/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 440, in execute
2023-09-02 13:39:15     self.error_handler.check_response(response)
2023-09-02 13:39:15   File "/usr/lib/python3.11/site-packages/selenium/webdriver/remote/errorhandler.py", line 245, in check_response
2023-09-02 13:39:15     raise exception_class(message, screen, stacktrace)
2023-09-02 13:39:15 selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//button[@type='submit'][contains(text(), 'ログイン') or contains(text(), 'Login') or contains(text(), '登录') or contains(text(), '로그인') or contains(text(), '登入')]"}

Refactoring project

The project structure may need to be refactored.

Add log for debug

Import format

Optimize import format.

Provide an interface to download illustrations from pixiv

Give an illustration id, and use this id to get a download link for this image.

Currently getIllustListByUid can only get the url of all illustrations for a specific user. because these illustrations may involve r18 content, the download link is not given directly. Instead, it is a preview link that requires permission.

Using Nginx to reverse proxy flask

see -> NGINX Reverse Proxy