Comments (4)
Sure, I am open to the idea. Most of the code was written quite a while ago. Did you have any particular language features/constructs in mind?
from kijiji-scraper.
For starters, replacing var w/ const/let following this guideline https://eslint.org/docs/rules/no-var. And maybe less reliance on jquery. I think it would be more of a refactor than anything lol.
from kijiji-scraper.
This module is already pretty small, but I agree that some refactoring would be beneficial. The code is currently very procedural and could stand to improve by becoming more object-oriented. The most glaring example that I can think of is ad-parser.js. An Ad
object could be created and the functions defined in ad-parser.js could instead be defined as member functions of Ad
.
Additionally, the new Ad
object could be returned by both the RSS feed scraper (kijiji-query.js) and regular scraper (ad-scraper.js), increasing consistency. No more ad
/innerAd
weirdness (the scapeInnerAd
option could still be supported via lazy initialization upon first access to ad information).
As for the reliance on cheerio (jQuery-like library), I'm not sure how this would work well without it. We'd be re-implementing all of their great work. Use of that library provides an easy way to read Kijiji's markup and is one of the reasons the code for this module isn't huge.
If you'd like to open a PR for some of this, I'm all for it. I will work on improving this project it in my spare time either way, but if we coordinate then we won't duplicate our efforts. 😄
from kijiji-scraper.
Good news!
I've had a lot of ideas for ways to improve this module for a while which were on the back burner due to life, etc. You gave me some motivation to get back to it. I've refactored this module to be more object-oriented, fixing some things that have bugged me and adding features while I was at it.
ES6 features now being used:
- Replaced
var
withlet/const
- Arrow functions
- Template literals
- Default parameters
- ES6-style class definitions
- Native promises
Refactored:
- New
Ad
class to handle scraping and converting ads to strings Ad
objects returned from searching are now the same as the ones created by scraping a single ad (consistency at last!)
New features:
- Can now specify minimum/maximum number of search results. The scraper will automatically get search result pages until the number has been reached
- Helper objects have been defined to make using
locationId
andcategoryId
easier (see the readme andlib/locations.js
/lib/categories.js
) - Promise support (callbacks are still supported as optional parameters)
Check the readme for a more detailed explanation :)
from kijiji-scraper.
Related Issues (20)
- Random Error:Invalid Kijiji HTML on search results page HOT 5
- Issues with search for services? HOT 7
- Scraper fails to return search results (Kijiji scraper detection + blocking) HOT 23
- Search() with URL HOT 1
- "npm install" fails HOT 9
- MinResults and MaxResults in Search() function implemented improperly, Doesn't allow for getting every listing available. HOT 9
- kijiji.search() Randomly Returns an Amount of Ads Under Specified Amount via minResults HOT 4
- Consideration for expanding repo to Kijiji API HOT 1
- Kijiji.Ad.Get() Doesn't return to callback HOT 1
- Impossible to fetch ads HOT 5
- Scraper fails to properly scrape some numeric attributes (real numbers) HOT 6
- Missing very new ads when using the Mobile API HOT 15
- Ad #... does not exist HOT 2
- Incorrect Size Attribute For Shoe Category HOT 6
- Category Attributes HOT 5
- Create helper types for enums, and expose expected response attributes
- Package broken: Kijiji-scraper returns no results HOT 5
- Not working HOT 5
- Kijiji Mingle API Issue HOT 11
- How long is a temporary block? I haven't been able to get results since yesterday. HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kijiji-scraper.