yad-2-puppeteer-rental-scraper's Introduction

Yad-2 Scraper with Puppeteer

**Adapted By Sara Weissman**

This project is open for collaboration.

The project aims to scrape apartment rental prices from Yad-2 (Hebrew IL) site: https://www.yad2.co.il/realestate/rent

Data will be collected from class="feeditem" and will exclude the top table of class="feed-list-platinum".

Technologies used:

[Node.js] - backend
[VUE.js] - frontend
[MongoDB] - database

Some excellent information on scraping with puppeteer in the following links:

Known Issues:

Adapting Puppeteer scraper for all URLs on the site:

Looping through URLs + looping through pagination:

Issues with looping through both of the above loops simultaneously (might be solved with Puppeteer Cluster)
Issues with looping when pagination div doesn’t exist on page (might be solved with try-catch)

Method for scraping data from ex. Tel-Aviv area needs re-writing, such as: see Puppeteer Docs

Data Accuracy Issues:

Sub-areas: refine data by price per neighborhood

Screenshots

yad-2-puppeteer-rental-scraper's People

Contributors

Watchers

yad-2-puppeteer-rental-scraper's Issues

Looping simultaneously through URLs and pagination

URLs loop:

for (const url of GETURLS) { ... }

paginating has issue's with some URLs:

let isBtnDisabled = false;
      while (!isBtnDisabled) {
const is_disabled =
          (await page.$(".pagination-nav.text.disabled")) !== null;
        isBtnDisabled = is_disabled;
        if (!is_disabled) {
          await Promise.all([
            page.click(".next-text"),
            page.waitForNavigation({ waitUntil: "networkidle2" }),
          ]);
        }
      }

Recommend Projects