Comments (4)
GoogleCodeExporter
commented on August 15, 2024
Sorry mate, this is a basic requirement which I don't see how we can solve...
Original comment by [email protected] on 18 Jan 2015 at 5:48
- Changed state: Accepted
from crawler4j.
GoogleCodeExporter
commented on August 15, 2024
Not a bug, you might want to move this thread to the forum though...
Original comment by [email protected] on 19 Jan 2015 at 4:31
- Changed state: Invalid
from crawler4j.
GoogleCodeExporter
commented on August 15, 2024
Where is the forum? Also, how difficult would it be if I try to move from
Berkeleydb to mysql?
Original comment by [email protected] on 19 Jan 2015 at 4:43
from crawler4j.
GoogleCodeExporter
commented on August 15, 2024
Forum: https://groups.google.com/forum/#!forum/crawler4j
It is totally doable, and there is even a card to change crawler4j so any one
will be able to incorporate his own type of internal storage.
But it won't be so easy, and if you want to have your mysql on an external
server then the crawler might be very very slow...
Original comment by [email protected] on 20 Jan 2015 at 8:35
from crawler4j.
Related Issues (20)
- HttpResponse response = httpClient.execute(get) in PageFetcher has no Timeout HOT 2
- Change logging from System.out.print to SLF4J HOT 1
- WebCrawler should throw exceptions instead of returning at the middle of the method HOT 2
- Upgrade CustomFetchStatus HOT 1
- NullPointerException when crawling links with no HREF HOT 1
- emove Non official http status codes HOT 1
- PageFetcher is unreadable HOT 1
- crawler fail due to http 303 see other HOT 8
- Scraping iframes, base64,vb scripts HOT 6
- FileNotFoundException: .m2\repository\edu\uci\ics\crawler4j\4.0\crawler4j-4.0.jar!\tld-names.zip HOT 8
- Proxy information get lost when using basic authentication
- Slf4j libraries missing on some configurations HOT 1
- Jar with dependencies
- Crawling over disallowed paths from robots.txt HOT 1
- robots.txt isn't crawled HOT 4
- Hanging on file process HOT 4
- Memory usage
- Crawler time delay when exiting
- Crawler is slow
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from crawler4j.