Impossible to fetch ads about kijiji-scraper HOT 5 CLOSED

Hm. How are you running the code? It looks like it's in a browser, that could be why (although I would expect that to work).

Could you please provide the details of the environment you are running this within and full sample code that reproduces the problem?

Thanks

from kijiji-scraper.

Hello,
Thanks for your help, indeed I tried to execute this code front side.

I tried in a NextJS app (version 10.0.6) and into a create-react-app boilerplate, I just generated a new app with npx create-react-app my-app and in both cases I had the error above

from kijiji-scraper.

I was able to reproduce the issue in a simple project generated with create-react-app. Using Firefox, I received a slightly more informative error:

TypeError: 'fetch' called on an object that does not implement interface Window.

It turns out that the library I'm using for network communication (node-fetch) does not work on the client side. It is self-described as "minimal code for a window.fetch compatible API on Node.js runtime" (i.e., not browsers).

There are other libraries, such as cross-fetch which do environment detection at runtime and use either window.fetch or node-fetch as appropriate. This allows using the same API regardless of where the code is running. However, even if I change libraries, there is still a problem:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://mingle.kijiji.ca/api/ads/1553070389. (Reason: CORS request did not succeed).

(1553070389 is just an arbitrary ad I chose).

The problem is that Kijiji does not set the Access-Control-Allow-Origin CORS HTTP header which means that even when the proper library is used, scripts running in browsers will not be able to successfully complete the request due to the same-origin policy.

So in order to get around this, I suggest that you move your Kijiji scraping logic to the server side (Node.js), since it is not bound by the same-origin policy. You can create a server endpoint in your app that acts as a proxy. The client can send it an ad URL as input and the endpoint handler can use the scraper to retrieve the information from Kijiji and send it back to the client.

from kijiji-scraper.

Thanks for your explanation, this is exactly what I’ve done and it works perfectly

from kijiji-scraper.

Perfect, glad to hear it! I'm going to close this then. All the best.

from kijiji-scraper.