kijiji.search() Randomly Returns an Amount of Ads Under Specified Amount via minResults about kijiji-scraper HOT 4 CLOSED

Thanks for the clear reproduction steps. This looks like a pretty simple problem.

To detect whether or not the module has received all of the results, I check to see if the current page of results contains a link to the next page. If there is no next page link, we have received all of the results. This is done using a regex. But sometimes the result XML has:

<types:link href="http://link-to-next-page" rel="next"/>

and other times it has

<types:link rel="next" href="http://link-to-next-page"/>

The regex can't handle the different ordering of tags. I'll switch it to actually parse the XML.

from kijiji-scraper.

That's why you aren't supposed to parse XML/HTML with Regex. See first answer

Jokes aside, you may want to switch to an XML parsing library to avoid stuff like this from occurring in the future, but only if you think it will happen again. Otherwise it's a waste of time. Thanks for the bug fixes!

from kijiji-scraper.

Haha that's right. In this case it seemed good enough, but it looks like I was wrong! Never trust consistency in foreign input.

I'm already using an HTML+XML parsing library (cheerio) for everything else in this module, so it makes sense to use it for this too. Now everything is more consistent.

Try with the latest (6.1.1).

from kijiji-scraper.

Works well!

from kijiji-scraper.