Crawl JS generated docs about docsearch-scraper HOT 10 CLOSED

Note to myself:
Add search on gns3: https://secure.helpscout.net/conversation/154616922/4890/?folderId=696715

from docsearch-scraper.

This looks like a pretty big enhancement, considering that the underlying engine we're using for scrapping (Scrapy) only do static HTML parsing. For SPA application, I would rather try to hit the API level if possible, or say that DocSearch is not compatible with their documentation.

For Prezly, they are using readme.io, maybe we could create something directly on readme.io level

from docsearch-scraper.

I'd say it's more a feature than an enhancement.

I would personally go with an optional HTTP Proxy which can process JavaScript (PhantomJS / Selenium) documentations and feed the resulting static page into Scrapy / Python. What do you think about this approach?

from docsearch-scraper.

There is Scrapy for JS: https://github.com/scrapinghub/scrapy-splash

from docsearch-scraper.

@ElPicador very cool! As I can see it's basically what I said, just more handy and already Dockerized 😄 did you already give it a try?

from docsearch-scraper.

Never, @redox was the one who told me about it

from docsearch-scraper.

@ElPicador @proudlygeek @pixelastic We've been thinking of making it the onboarding project of @aseure :)

from docsearch-scraper.

Awesomeness!!! 💯 👍

from docsearch-scraper.

I've opened a PR to address those problematic documentations. Please see #46.

from docsearch-scraper.

I think this can be closed

from docsearch-scraper.