Comments (9)
To solve the issue, just add a throw in public FetchedData fetch(Resource resource) throws Exception, and also add a try/catch in public FetchedData next() around the fetch() function. Cheers
from sparkler.
This is a good fix. I understood that you want to remove the pain of handling such errors on modules. 👍
from sparkler.
@arelaxend Thanks for reporting this issue and also for a quick solution. I will investigate more before putting it around try and catch.
Just a suggestion - If you know what URLs are causing this issue and you don't want to crawl them as well, place URL regex filter(s). It will save you some time. :)
from sparkler.
Thanks @arelaxend If you could submit a pull request with those fixes, it will be awesome 👍 🥇
from sparkler.
@karanjeets
Instead of
Just a suggestion - If you know what URLs are causing this issue and you don't want to crawl them as well, place URL regex filter(s). It will save you some time. :)
I suggest we do this:
To solve the issue, just add a throw in public FetchedData fetch(Resource resource) throws Exception, and also add a try/catch in public FetchedData next() around the fetch() function.
from sparkler.
Hi, I got:
fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.
Besides, you just have to update the function fetch() in the plugin fetcher-jbbrowser. I choose to do that to be coherent with what you have done in the fetch() function on the app folder. 👍 And so, it means to change the implementation of the fetch() function by the following code:
public FetchedData fetch(Resource resource) {
LOG.info("JBrowser FETCHER {}", resource.getUrl());
FetchedData fetchedData;
/*
* In this plugin we will work on only HTML data
* If data is of any other data type like image, pdf etc plugin will return client error
* so it can be fetched using default Fetcher
*/
try {
if (!isWebPage(resource.getUrl())) {
LOG.debug("{} not a html. Falling back to default fetcher.",
resource.getUrl());
//This should be true for all URLS ending with 4 character file extension
//return new FetchedData("".getBytes(), "application/html", ERROR_CODE) ;
return super.fetch(resource);
}
long start = System.currentTimeMillis();
LOG.debug("Time taken to create driver- {}",
(System.currentTimeMillis() - start));
// This will block for the page load and any
// associated AJAX requests
driver.get(resource.getUrl());
int status = driver.getStatusCode();
//content-type
// Returns the page source in its current state, including
// any DOM updates that occurred after page load
String html = driver.getPageSource();
//quitBrowserInstance(driver);
LOG.debug("Time taken to load {} - {} ", resource.getUrl(),
(System.currentTimeMillis() - start));
if (!(status >= 200 && status < 300)) {
// If not fetched through plugin successfully
// Falling back to default fetcher
LOG.info(
"{} Failed to fetch the page. Falling back to default fetcher.",
resource.getUrl());
return super.fetch(resource);
}
fetchedData = new FetchedData(html.getBytes(), "application/html",
status);
resource.setStatus(ResourceStatus.FETCHED.toString());
fetchedData.setResource(resource);
return fetchedData;
} catch (Exception e) {
LOG.info(
"{} Failed to fetch the page. Falling back to default fetcher.",
resource.getUrl());
return super.fetch(resource);
}
}
from sparkler.
Plus, for the selenium error, it happens when you build the app, but the jar file for the plugins are not generated! 👍
from sparkler.
Thanks for the comment
Hi, I got:
fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.
What command did you execute to get this error message?
I hope you are aware of the way to raise pull request without having write permissions. If not, refer to http://stackoverflow.com/a/14681796/1506477. Basically, you (1) fork this repo (2) push your changes to your fork (3) raise a pull request from your fork to this repo.
Plus, for the selenium error, it happens when you build the app, but the jar file for the plugins are not generated!
Ah! That explains the NullPointerException - the 3rd stack trace.
from sparkler.
Fixed in #61
Waiting for one more person to review before I merge
from sparkler.
Related Issues (20)
- Update CI so users can download built Sparkler package
- Investigate pipeline frameworks HOT 1
- Sparkler cannot be executed on Databricks because sparkContext not pulled from sparkSession
- Sparkler Elasticsearch storage engine HOT 10
- Elasticsearch for Sparkler - Command Line Configuration HOT 2
- Elasticsearch for Sparkler - Containerization Logic HOT 4
- Elasticsearch for Sparkler - Maven Profiles
- Elasticsearch for Sparkler - Factory Design Pattern HOT 1
- Broken run script HOT 4
- Fix sparkler CI build
- Writing Data to Elasticsearch Storage Engine HOT 1
- Failed to create thread
- Unit Tests for Sparkler and Elasticsearch
- Debugging Elasticsearch Connection
- Sparkler not distributing work over nodes HOT 1
- Exclude net.jpountz.lz4 lz4 from kafka-clients dependency in sparkler-app/pom.xml HOT 2
- Caught Server refused connection at: http://localhost:8983/solr/crawldb HOT 19
- Build fails: could not find com.browserup:browserup-proxy-core:jar:3.0.0-SNAPSHOT
- Error from server at http://localhost:8983/solr/crawldb: ERROR: [doc=<>] unknown field 'contenthash' HOT 3
- warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sparkler.