Comments (4)
We already provide a universal parsing method via cheerio, all crawler types have a parseWithCheerio
context helper for that.
As I mentioned, if you have some library for XPATH support on your mind that we could wrap, we can surely do that.
from crawlee.
Puppeteer and Playwright support xpath selectors (https://playwright.dev/docs/locators#locate-by-css-or-xpath)
At a quick glance, I don't see cheerio supporting it, but maybe we can potentially look into it for cheerio too
from crawlee.
As Vlad already mentioned, this is supported natively in browsers and you can use that. Crawlee does not "support selectors", it only abstracts use of other libraries like cheerio that handle things like that. I will close this as it feels not actionable, if you have some library on your mind that you'd like us to wrap, that's a different story.
At a quick glance, I don't see cheerio supporting it, but maybe we can potentially look into it for cheerio too
That feels outside of scope, and looking at the cheerio issues this idea got rejected on their end too.
from crawlee.
Similar to Scrapy's parsing support, which remains independent of whether you use playwright, puppeteer, or any other method to fetch HTML, having a universal parsing method is a great step. Ultimately, we're dealing with HTML that needs to be parsed, so the method of obtaining it, whether through playwright or puppeteer, shouldn't determine our parsing approach.
Apologies if I came across as nerdy. 🙏
from crawlee.
Related Issues (20)
- `maxUsageCount: 1` does not retire session after a single use HOT 1
- `useIncognitoPages` doesn't rotate fingerprints HOT 1
- Add support for all tags defined by the sitemap protocol
- `page.evaluate` results error HOT 2
- HttpCrawler - determining character encoding
- Add `waitForAllRequestsToBeAdded` option to `enqueueLinks`
- Multiple calls to enqueueLinks with Promise.all result in a crash HOT 2
- `RestrictedCrawlingContext` should not extend `Record<string, unknown>` HOT 2
- Could not kill browser: Cannot read private member #process from an object whose class did not declare it HOT 2
- Image not available(build status) in readme
- page.waitForTimeout is removed
- `querySelector` should be propagated to all crawling contexts HOT 2
- Write an e2e test of adaptive playwright crawler
- Adaptive crawling
- Initial PoC version of adaptive crawler
- ProxyUrl not accepted: "(array `proxyUrls`) Expected property string values to be a URL, got " HOT 1
- Detect when user calls global SDK methods from "side-effect free" context HOT 1
- Tor as proxy
- Adopt a code formatter and enforce it with CI
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from crawlee.