jtiala / wpdl Goto Github PK

Might be worth converting the repo to TS while at it.

Would be sweet to run the script with npx wpdl

`root@DREAM-PC:/mnt/c/Users/SenHan/Desktop/wpdl# npx wpdl --url https://xxx.xxx --posts --removeEmptyElements
--- wpdl ---

Starting to scrape https://xxx.xxx ...

Scraping posts...

Scraping page 1 of 3368 from https://xxx.xxx/wp-json/wp/v2/posts?order=desc ...

Scraping post 139828-momo-sakura-%e6%a1%9c%e7%a9%ba%e3%82%82%e3%82%82-%e3%82%a2%e3%82%b5%e8%8a%b8sexy%e5%a5%b3%e5%84%aa%e5%86%99%e7%9c%9f%e9%9b%86-%e3%80%8c%e5%a5%bd%e3%81%8d%e3%80%81%e3%81%a7%e3%81%99%e3%80%82%e3%80%8d-4...
Found 1 image(s).
file:///C:/Users/SenHan/Desktop/wpdl/src/utils/scraping.js:207
mediaItem.media_details.sizes.full.source_url &&
^

TypeError: Cannot read properties of undefined (reading 'source_url')
at downloadMediaItemImage (file:///C:/Users/SenHan/Desktop/wpdl/src/utils/scraping.js:207:40)
at downloadImages (file:///C:/Users/SenHan/Desktop/wpdl/src/utils/scraping.js:199:13)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async dataHandler (file:///C:/Users/SenHan/Desktop/wpdl/src/scrapers/posts.js:125:11)
at async paginatedScrape (file:///C:/Users/SenHan/Desktop/wpdl/src/utils/scraping.js:58:5)
at async scrapePosts (file:///C:/Users/SenHan/Desktop/wpdl/src/scrapers/posts.js:44:3)
at async file:///C:/Users/SenHan/Desktop/wpdl/src/wpdl.js:191:3

Node.js v18.14.0
root@DREAM-PC:/mnt/c/Users/SenHan/Desktop/wpdl#`

I was interested in seeing how this tool compares to wp-json-scraper, but while it worked for one of the sites I've used wp-json-scraper on, it didn't work on another.

The site it's not working on is https://powershellisfun.com/

Sidenote: In your README.md you can remove the bit about npm run scrape -- since npx is bundled with npm nowadays.