Article page is not compatible with pocket about manshar HOT 6 CLOSED

From what I read in the readme file, this is a single page app. It uses javascript to ajax-request actual content of the article.

Pocket is probably using some kind of html scraping to get the content from the first response, and the first response of this app will have nothing but javascript and the basic templates.

You can confirm this by disabling javascript and reloading the article page.

from manshar.

Exactly as @tolbahadi explained - For the Pocket case, you can save the article in pocket, except that it wouldn't have a proper title/description.

Maybe there should be a fallback rails route that would check for the useragent string, and when appropriate render a plain HTML file?

from manshar.

That is a good solution @zakelfassi +1

from manshar.

Unfortunately I just spent some time trying to figure out if Pocket sends a special request that we can detect. We already have a fallback in place for all other crawlers (Google, Facebook, Twitter...etc) and I just sent PR #222 to detect more bots (incl. Slack bots).

I've done some debugging and it doesn't seem Pocket sends anything special with the request, they don't use the ?_escaped_fragment_= recommendation or use a custom user-agent to fetch. I've contacted them and hoping to hear from them.

If anybody has any insights in this please let us know :-)

from manshar.

btw, heard back from Pocket support they might add us to a whitelist to crawl our content using the ?_escaped_fragment_= way. Not sure how long will that take to be in place. They also told me they're working on fixing this by developing a better detection mechanism for pages that load content through ajax.

from manshar.

I'll be closing this one as there is no easy way to fix this without Pocket help. However, pocket support told me that they're working on better crawling of ajax-based sites.

from manshar.