Comments (5)
First, thank you for taking the time to write out a detailed description with code snippets and error messages. One thing to note though is that you have your categoryId
and locationId
switched. Your prefs
object should instead be defined as:
var prefs = {
"locationId": 1700185,
"categoryId": 27
}
if you have an incorrect locationId
or categoryId
, the ads
array will be empty. However, it looks like you're having different problems.
On to your issue:
I am not familiar with Laravel. What type of directory structure do you have (I presume you are running npm run
from your project's top-level directory)? Also, which script from your package.json
's scripts
object are you trying to run, and could you post it here? The problem is likely with your configuration.
On my test machine, I am able to run (where app.js
is the sample code you posted with my changes to the prefs
object mentioned above)
$ npm install kijiji-scraper
$ node app.js
and the script prints the array of search results, as expected.
from kijiji-scraper.
thanks for trying to help!
I am unfortunately not all that familiar with running pure node apps. I actually tried to set it up with https://github.com/sahat/hackathon-starter but I was completely at a loss on where to even add the kijiji-scraper code
I would really like to get it to work in my laravel project because I would then be using laravel to store and analyze the scraped kijiji results and trigger notifications and other stuff from Laravel.
You are correct that I run 'npm run production or npm run watch' from the root directory.
The app.js is located in /resources/assets/js/
Is it possible this is causing an issue because your package is using relative paths?
Here's my package.json and and webpack.mix.js (used by Laravel Mix, similar to gruntfile)
{
"private": true,
"scripts": {
"dev": "cross-env NODE_ENV=development node_modules/webpack/bin/webpack.js --progress --hide-modules --config=node_modules/laravel-mix/setup/webpack.config.js",
"watch": "cross-env NODE_ENV=development node_modules/webpack/bin/webpack.js --watch --progress --hide-modules --config=node_modules/laravel-mix/setup/webpack.config.js",
"hot": "cross-env NODE_ENV=development node_modules/webpack-dev-server/bin/webpack-dev-server.js --inline --hot --config=node_modules/laravel-mix/setup/webpack.config.js",
"production": "cross-env NODE_ENV=production node_modules/webpack/bin/webpack.js --progress --hide-modules --config=node_modules/laravel-mix/setup/webpack.config.js"
},
"dependencies": {
"axios": "^0.15.2",
"bootstrap": "^3.0.0",
"cross-env": "^3.2.3",
"jquery": "^2.1.4",
"js-cookie": "^2.1.0",
"kijiji-scraper": "^2.0.0",
"laravel-mix": "0.*",
"moment": "^2.10.6",
"promise": "^7.1.1",
"sweetalert": "^1.1.3",
"underscore": "^1.8.3",
"urijs": "^1.17.0",
"vue": "2.*"
}
}
let mix = require('laravel-mix');
var path = require('path');
/*
|--------------------------------------------------------------------------
| Mix Asset Management
|--------------------------------------------------------------------------
|
| Mix provides a clean, fluent API for defining some Webpack build steps
| for your Laravel application. By default, we are compiling the Sass
| file for the application as well as bundling up all the JS files.
|
*/
mix.less('resources/assets/less/app.less', 'public/css')
.copy('node_modules/sweetalert/dist/sweetalert.min.js', 'public/js/sweetalert.min.js')
.copy('node_modules/sweetalert/dist/sweetalert.css', 'public/css/sweetalert.css')
.js('resources/assets/js/app.js', 'public/js')
.webpackConfig({
resolve: {
modules: [
path.resolve(__dirname, 'vendor/laravel/spark/resources/assets/js'),
'node_modules'
],
alias: {
'vue$': 'vue/dist/vue.js'
}
}
});
I've added the code inside the app.js
any ideas?
from kijiji-scraper.
Ah, okay, I think I see what's wrong. I've looked into Laravel a little bit and it is a server-side technology (PHP). It looks like you are attempting to run kijiji-scraper on the client-side (you are putting the code that uses it in the public/js
directory). Also notice that all of the dependencies in your package.json
are client-side JS libraries (i.e., for use in the browser - not on the server; except laravel-mix, which is just used to compile your assets into one bundle, if I'm not mistaken).
The scraper must run on the server-side under Node.js, unless you use something like Browserify, which Laravel happens to support through its "Elixir" (see its documentation, specifically the Browserify section). Using this on your app.js
(or another script with uses the scraper), you should be able to use kijiji-scraper in the browser.
from kijiji-scraper.
again huge thanks for explaining!
i don't use elixir.. no one does anymore.. it might be possible to do what you say with Laravel Mix.. but maybe you're just not supposed to add server-side js the way I was tyring to do it.
So instead I made a blank app.js in the root and added the code there and indeed after doing node app.js it just works!
I am now having my Laravel application call the node script, grab the output and then process it. But I'm running into another issue because of invalid json which I opened another issue about
It would also be useful if there was a way of getting more or less than the default 20 entries
And what about other parameters such as radius ?
Thanks!!
from kijiji-scraper.
Good to hear you found a workaround. I'll close this issue.
If you're using the same script as in your original post (or a similar one) to output the ad information, then the JSON issue is not with the library. I posted more information over in #8.
The query/search functionality uses Kijii's RSS feed functionality, which returns 20 entries per page. Paging is currently not implemented in the scraper. The RSS feed seems to support it (e.g., https://www.kijiji.ca/rss-srp-ottawa/cars/page-2/k0l1700185) but using it in a nice way (without using strings like "cars" and "Ottawa" in place of locationId
and categoryId
) was flaky when I played around with it just now. I'd need to research it more, or switch over to scraping the HTML search results page. That page has trivial paging via the bar at the bottom but the scraper is more likely to break if it is used since the HTML layout can change when Kijiji updates their UI (this recently happened with ad pages, see issue #6). In the mean time, PRs are welcome.
The way I find other search parameters is by performing a search on Kijiji and using my browser's developer tools (F12 key). The "network" tab shows network requests and what parameters are sent. Playing around with different search parameters and observing the requests made to Kijiji can reveal their names. Sidenote though: it looks like Kijiji has changed the way they do parameters on their site. For example, after setting the radius to 100km, the URL for the search results page had r100.0
appended to the query string instead of as a POST parameter. The old method (used by this library) of using custom parameters should still work though. Please let me know if you experience problems with it.
from kijiji-scraper.
Related Issues (20)
- Is it possible to add several recipients emails addresses ? HOT 2
- When scrapeResultDetails is set to false, listing Titles comeback as empty Strings HOT 4
- Random Error:Invalid Kijiji HTML on search results page HOT 5
- Issues with search for services? HOT 7
- Scraper fails to return search results (Kijiji scraper detection + blocking) HOT 23
- Search() with URL HOT 1
- "npm install" fails HOT 9
- MinResults and MaxResults in Search() function implemented improperly, Doesn't allow for getting every listing available. HOT 9
- kijiji.search() Randomly Returns an Amount of Ads Under Specified Amount via minResults HOT 4
- Consideration for expanding repo to Kijiji API HOT 1
- Kijiji.Ad.Get() Doesn't return to callback HOT 1
- Impossible to fetch ads HOT 5
- Scraper fails to properly scrape some numeric attributes (real numbers) HOT 6
- Missing very new ads when using the Mobile API HOT 15
- Ad #... does not exist HOT 2
- Incorrect Size Attribute For Shoe Category HOT 6
- Category Attributes HOT 5
- Create helper types for enums, and expose expected response attributes
- Package broken: Kijiji-scraper returns no results HOT 5
- Not working HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kijiji-scraper.