Comments (4)
Great Issue. Here is my proposal -
- Create a seperate schema to store cookies. Schema should have hostname as id
- Storing cookies should be an additional functionality which can be turned off through configuration
- Create utility script/app for users to add cookies to schema. These cookies should never be overridden/updated during crawl
Thoughts?
+@sujen1412
from sparkler.
Sounds like a great plan.
Agreed, we will need a storage to preserve cookies.
We need storage to keep robots.txt and thus resolve - #45
We need to have a discussion with all of us in team ( @karanjeets @sujen1412 ) before we jump in.
Also please have a look at #45, you may see that they are kind of similar and thus the solution can be designed with a broader scope
from sparkler.
@smadha we should flesh out the details a little more here.
- How will a user provide the cookies ?
- How will Sparkler store the cookies ?
- What is the mechanism to let sparkler know on what hosts to use the cookies ?
- How will Sparkler know that the cookie has expired/ or a new cookie is set by the server ? In this case should the crawling stop ? Should sparkler update the cookie in solr ? How will the user know about this ?
from sparkler.
All these are great points.
How will a user provide the cookies ?
M.S. - My thought is take a input file from user and process it with sparkler. Do you know about any preexisting format? If not we can define one which has a map of domain name to cookie value. Will be great if you can share your own file, I think you said you have a custom way for processing cookies in nutch.
How will Sparkler store the cookies?
M.S. My proposal is to store in a solr schema. We should define a concrete schema before we implement, but my idea was to maintain one solr doc for each domain/subdomain name.
What is the mechanism to let sparkler know on what domains to use the cookies ?
M.S. If cookie processing is enabled we will search in our schema for available cookies for that domain/subdomain
How will Sparkler know that the cookie has expired/ or a new cookie is set by the server ? In this case should the crawling stop ? Should sparkler update the cookie in solr ? How will the user know about this ?
M.S. Great point. My proposal is to mimic browser behaviour for cookies. We should put expiry check while searching for cookies in solr.
- We can put cookie info in debug logs. In my opinion we should not stop crawl, if there are no valid cookies we should continue to crawl without cookies.
- We can add more logs for cookies manually added by user if that helps. Like log whole cookie if it's picked up or else log no cookies found. We can also log solr search queries so that people can debug if needed.
Will be great if you can demo your current approach to us.
Thanks
from sparkler.
Related Issues (20)
- Update CI so users can download built Sparkler package
- Investigate pipeline frameworks HOT 1
- Sparkler cannot be executed on Databricks because sparkContext not pulled from sparkSession
- Sparkler Elasticsearch storage engine HOT 10
- Elasticsearch for Sparkler - Command Line Configuration HOT 2
- Elasticsearch for Sparkler - Containerization Logic HOT 4
- Elasticsearch for Sparkler - Maven Profiles
- Elasticsearch for Sparkler - Factory Design Pattern HOT 1
- Broken run script HOT 4
- Fix sparkler CI build
- Writing Data to Elasticsearch Storage Engine HOT 1
- Failed to create thread
- Unit Tests for Sparkler and Elasticsearch
- Debugging Elasticsearch Connection
- Sparkler not distributing work over nodes HOT 1
- Exclude net.jpountz.lz4 lz4 from kafka-clients dependency in sparkler-app/pom.xml HOT 2
- Caught Server refused connection at: http://localhost:8983/solr/crawldb HOT 19
- Build fails: could not find com.browserup:browserup-proxy-core:jar:3.0.0-SNAPSHOT
- Error from server at http://localhost:8983/solr/crawldb: ERROR: [doc=<>] unknown field 'contenthash' HOT 3
- warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sparkler.