Support JavaScript generated Websites about rss-proxy HOT 5 CLOSED

A short update on this issue. As mentioned in this PR backing JavaScript support into rss-proxy breaks some architectural concepts. Moreover, some dynamic websites often use click callbacks and don't follow the semantics of html tags like this one https://bulletin.nu/. So, for this edge case, to make rss-proxy work with these kind of sites, a JavaScript rendering engine would need to mark the click relevant nodes, that rss-proxy can understand it. Then rss-proxy could easily be extended to look for those nodes as well.

from rss-proxy.

I finalized the a basic support for dynamic websites. The functionality is available in a separate image damoeb/rss-proxy:js, cause it is larger than 1gb due to the browser dependencies, the rss-proxy layer is around 40mb.

docker pull damoeb/rss-proxy:js
docker run -p 3000:3000 -it damoeb/rss-proxy:js

In the logs you should than see then info: + Enabling JavaScript statement. This will activate a checkbox in the UI, where you can toggle dynamic rendering in a headless chrome.
I tested it with https://www.asus.com/Networking-IoT-Servers/WiFi-Routers/ASUS-WiFi-Routers/RT-AC5300/HelpDesk_BIOS/ which was requested in issue 11.

from rss-proxy.

i tried with rss-proxy:js and when javascript is enabled, no site loads on right panel. tried in multiple browser, no luck

from rss-proxy.

was finally able to get the site to load, then added the feed to tt-rss. got this error

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
    <id>tag:rss-proxy.migor.org:error-contextXPath://*[@id='Manual-Download']/div[2]/div[2]/div[1]/div[1]/section[1]/div-linkXPath:./div[2]/a[1]</id>
    <title>rss-proxy error</title>
    <updated>2021-01-05T00:43:11.449Z</updated>
    <generator>rss-proxy</generator>
    <link rel="alternate" href="https://www.asus.com/us/Networking/RT-AC5300/HelpDesk_BIOS/"/>
    <rights>rss-proxy</rights>
    <entry>
        <title type="html"><![CDATA[Error]]></title>
        <id>https://rssproxy.migor.org?url=https%3A%2F%2Fwww.asus.com%2Fus%2FNetworking%2FRT-AC5300%2FHelpDesk_BIOS%2F</id>
        <link href="https://rssproxy.migor.org?url=https%3A%2F%2Fwww.asus.com%2Fus%2FNetworking%2FRT-AC5300%2FHelpDesk_BIOS%2F"/>
        <updated>2021-01-05T00:43:11.449Z</updated>
        <content type="html"><![CDATA[Unable to download http://localhost:3000/api/dynamic?url=https://www.asus.com/us/Networking/RT-AC5300/HelpDesk_BIOS/, cause null Try to resolve the issue on provided link.]]></content>
        <published>2021-01-05T00:43:11.449Z</published>
    </entry>
</feed>

am self hosting, other non js rss-proxy feeds works fine with tt-rss

from rss-proxy.

Looking into it :)

from rss-proxy.