Comments (5)
It appears that patreon api response time really slows down after reaching certain page (for this creator it happens on page 41-42).
I will need to add some timeout management code into page crawling process to fix this. Will look into it when I have some free time.
from patreondownloader.
Interestingly enough this doesn't appear to happen on every creator who have a lot of pages. Removing this from 0.10.0 milestone for now since this needs further investigation
from patreondownloader.
Why not just increase the timeout? Or make it adjustable?
from patreondownloader.
For everyone else stumbling upon this here is a patch that just increases the timeout and works around this problem:
diff --git a/PatreonDownloader.Engine/WebDownloader.cs b/PatreonDownloader.Engine/WebDownloader.cs
index 9b998d5..022af01 100644
--- a/PatreonDownloader.Engine/WebDownloader.cs
+++ b/PatreonDownloader.Engine/WebDownloader.cs
@@ -87,7 +87,7 @@ namespace PatreonDownloader.Engine
{
IWebBrowser browser = await _puppeteerEngine.GetBrowser();
IWebPage page = await browser.NewPageAsync();
- await page.GoToAsync(url);
+ await page.GoToAsync(url, 1000*180);
string content = await page.GetContentAsync();
await page.CloseAsync();
from patreondownloader.
This should no longer happen in 0.10.0 since crawling process is no longer using puppeteer.
from patreondownloader.
Related Issues (20)
- Download files by month or tag
- Cookies issue HOT 1
- Browser doesnt open: Couldn't mmap icu data file HOT 1
- Fatal error on sending request HOT 2
- embedded videos aren't being downloaded (anymore?) HOT 1
- Campaign id cannot be less than 1 Error HOT 23
- Current user cannot view this post error message
- Preparing download and instant closes HOT 2
- FATAL Fatal error, application will be closed: UniversalDownloaderPlatform.Common.Exceptions.UniversalDownloaderException: Unable to retrieve crawl target info: Campaign id cannot be less than 1 (Parameter 'campaignId') HOT 8
- Downloading both a media and a post pic and they're identical HOT 1
- Support for external download link - cloud3.zoolz.com
- After logging in, Nothing happens HOT 4
- CampaignID Error HOT 1
- Chrome Page issue HOT 2
- fatal error
- Downloaded Images Display Artifacts HOT 1
- Cookie Errors
- ERROR Failed to download - Could not find part of the path, System Exception. HOT 2
- Downloader stops at 3900/ HOT 4
- cromium login problem HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from patreondownloader.