I was trying to use rendertron for the polymer sample shopping app.

when I test it locally it works fine and fast.

after I deploy, and try to visit it as a bot, I got this error

10 second time budget limit reached.
          Attempted rendering: http://whispering-bayou-43373.herokuapp.com/list/mens_outerwear
          Page load event fired: false

I'd like to know what makes it fail to detect the page load event.

PS: if you visit the page as a regular user everything is fine.

PPS: I removed all the pictures on purpose.

and

PoC: https://render-tron.appspot.com/screenshot/<h1>xss%20here<%2Fh1>%3Fwidth%3D2335%26height%3D1297

Allow the status code to be set within the response. Meta tag to start:

<meta name="bot-render-status" content="404">

Perhaps a JS API:

x.setStatusCode(404)

Everyone is saying that googlebot has no problem clawing polymer 2.0 websites.

according to polymer-project.org, it seems like a true statement.

I deployed a very simple polymer 2.0 app on heroku using express to serve the index.html.

and try to use google console to fetch the page.

Sadly, the result didn't show anything inside the shadow dom.

and currently googlebot is not on the user agent list.

Maybe we should add googlebot to the user agent list for rendertron middleware as well?

Allow for a config file.

cache: {
  enabled: true, 
  freshness: 1440
},

etc

By testing Rendertron, I came across a few sites rendertron is unable to render.

How about let's have a post for people to list the sites rendertron is not able to render?

I think it will make it easier for us to figure out what the pattern is and what needs to be fixed.

for example

https://mighty-dawn-10467.herokuapp.com/heroes

Angular4 + polymer 2.0 app based on the Angular 4 official tutorial Tour of Heroes

Note: I was able to use Rendertron to render this site before and let google/bing bots to successfully fetch the rendering result. It was perfect. But now Rendertron cannot even render it anymore.

https://whispering-bayou-43373.herokuapp.com/list/ladies_outerwear

Polymer 2.0 sample shopping app from Polymer CLI

Note: I was able to use npm start version of Rendertron to render this polymer 2.0 app. but now Rendertron cannot render it anymore.

ubuntu16.04 Docker version 17.03.1-ce, build c6d412e

When I tried

docker exec bot-render-container curl http://localhost:8080/?url=https://dynamic-meta.appspot.com

docker console output following logs

Listening on port 8080
{ Error: connect ECONNREFUSED 127.0.0.1:9222
at Object.exports._errnoException (util.js:1014:11)
at exports._exceptionWithHostPort (util.js:1037:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1138:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 9222 }

screenshot/https://fonts.google.com/?width=1280&height=1299 still hung after 85s. No errors, so something async never resolved.

First of all, I really appreciate your excellent work!

We've been thinking building a similar screenshot service after Chrome Headless come out, glad we found rendertron before we start developing our own.

The thing is, we intend to use this screenshot/render service in production environment but couldn't find any benchmark result or pressure test result, like how many screenshot requests can it take parallelly?

Do you have any plan doing this, how can we help out?

http://localhost:3000/?url=https://polymer-shop2.appspot.com&wc-inject-shadydom=true

Maybe something to do with the styles?

Adding an address without protocol gives
Eg : google.com
Cannot render google.com - "First argument must be a string, Buffer, ArrayBuffer, Array, or array-like object."

Maybe support for building the URL from the address provided ?

Instant of everyone need to build the image, it could be useful if we could just pull the image from Docker Hub or some other Docker Registry (quay? Google Cloud Registry?)

I have a notion that I'd like to add a cache layer that checks to see if the site has been rendered and stored in the last X amount of seconds. I have two ideas around this:

1. Use the google-cloud node libs to use datastore as a key/val store (ala https://github.com/stephenplusplus/google-cloud-kvstore)
2. Use an external redis/memcache setup (which is in the gae flex docs).

Option one obviously binds the container to be less portable (and I'd think this would be an enabled feature rather than a requirement to run the container). Option two has more flexibility (not bound to GCP), but still sits in the same boat (probably shouldn't be a requirement).

Would this be something that would be interesting to you @samuelli as a PR? Did you have any thoughts on the topic of caching?

Under consideration: Allow for a flag to notify when rendering is complete.

Need to find cases where the current loading function is insufficient. Do you have an example? Please comment below!

I was trying to use rendertron to render an Angular 4 app which is using polymer 2.0 web components. but rendertron simply ignores the shadow doms. (shadydom is set to true of course.)

the rendering result still has the web component tags like

but inside the tags, it's total empty.

this is the rendering result (I modified rendertron to console log the rendering result)

body: '<html lang="en"><head><!-- Shady DOM styles for dom-bind --><!-- Shady DOM styles for dom-repeat --><!-- Shady DOM styles for dom-if --><!-- Shady DOM styles for array-selector --><!-- Shady DOM styles for custom-style --><!-- Shady DOM styles for paper-ripple --><!-- Shady DOM styles for iron-overlay-backdrop --><!-- Shady DOM styles for iron-meta --><!-- Shady DOM styles for iron-dropdown --><!-- Shady DOM styles for fade-in-animation --><!-- Shady DOM styles for fade-out-animation --><!-- Shady DOM styles for paper-menu-grow-height-animation --><!-- Shady DOM styles for paper-menu-grow-width-animation --><!-- Shady DOM styles for paper-menu-shrink-width-animation --><!-- Shady DOM styles for paper-menu-shrink-height-animation --><!-- Shady DOM styles for paper-menu-button --><!-- Shady DOM styles for paper-radio-button --><!-- Shady DOM styles for iron-a11y-announcer --><!-- Shady DOM styles for iron-input --><!-- Shady DOM styles for paper-input-char-counter --><!-- Shady DOM styles for paper-input-container --><!-- Shady DOM styles for paper-input-error --><!-- Shady DOM styles for paper-input --><!-- Shady DOM styles for paper-button --><!-- Shady DOM styles for my-carousel -->\n <meta charset="utf-8">\n <title>Demo</title>\n <base href="/">\n\n\t\n\t\n\t\n\t\n\t\n\n\t\n\n\t<!-- <dom-module id="x-custom">\n\t <template>\n\t <button on-click="handleClick">Kick Me</button>\n\t </template>\n\t <script>\n\n\t //console.log(this);\n\n\t //this.mediator = require(\'assets/mediator/mediator.min.js\').Mediator;\n\t //this.theMediator = new Mediator();\n\n\t class XCustom extends Polymer.Element {\n\n\t static get is() {return \'x-custom\'}\n\n\t handleClick() {\n\t console.log(\'Ow!\');\n\t var event = new Event(\'build\');\n\n\t }\n\t }\n\t customElements.define(XCustom.is, XCustom);\n\t </script>\n\t</dom-module> -->\n\n <meta name="viewport" content="width=device-width, initial-scale=1">\n <link rel="icon" type="image/x-icon" href="favicon.ico">\n \n \n\n<custom-style><style type="text/css" is="custom-style">/* You can add global styles to this file, and also import other style files */\n</style></custom-style><custom-style><style is="custom-style"></style></custom-style><base href="http://localhost:4200/"></head>\n<body>\n <app-root _nghost-c0="">\n<div _ngcontent-c0="" style="text-align:center">\n <h1 _ngcontent-c0="">\n Welcome to </h1>\n <img _ngcontent-c0="" src="data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4NCjwhLS0gR2VuZXJhdG9yOiBBZG9iZSBJbGx1c3RyYXRvciAxOS4xLjAsIFNWRyBFeHBvcnQgUGx1Zy1JbiAuIFNWRyBWZXJzaW9uOiA2LjAwIEJ1aWxkIDApICAtLT4NCjxzdmcgdmVyc2lvbj0iMS4xIiBpZD0iTGF5ZXJfMSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayIgeD0iMHB4IiB5PSIwcHgiDQoJIHZpZXdCb3g9IjAgMCAyNTAgMjUwIiBzdHlsZT0iZW5hYmxlLWJhY2tncm91bmQ6bmV3IDAgMCAyNTAgMjUwOyIgeG1sOnNwYWNlPSJwcmVzZXJ2ZSI+DQo8c3R5bGUgdHlwZT0idGV4dC9jc3MiPg0KCS5zdDB7ZmlsbDojREQwMDMxO30NCgkuc3Qxe2ZpbGw6I0MzMDAyRjt9DQoJLnN0MntmaWxsOiNGRkZGRkY7fQ0KPC9zdHlsZT4NCjxnPg0KCTxwb2x5Z29uIGNsYXNzPSJzdDAiIHBvaW50cz0iMTI1LDMwIDEyNSwzMCAxMjUsMzAgMzEuOSw2My4yIDQ2LjEsMTg2LjMgMTI1LDIzMCAxMjUsMjMwIDEyNSwyMzAgMjAzLjksMTg2LjMgMjE4LjEsNjMuMiAJIi8+DQoJPHBvbHlnb24gY2xhc3M9InN0MSIgcG9pbnRzPSIxMjUsMzAgMTI1LDUyLjIgMTI1LDUyLjEgMTI1LDE1My40IDEyNSwxNTMuNCAxMjUsMjMwIDEyNSwyMzAgMjAzLjksMTg2LjMgMjE4LjEsNjMuMiAxMjUsMzAgCSIvPg0KCTxwYXRoIGNsYXNzPSJzdDIiIGQ9Ik0xMjUsNTIuMUw2Ni44LDE4Mi42aDBoMjEuN2gwbDExLjctMjkuMmg0OS40bDExLjcsMjkuMmgwaDIxLjdoMEwxMjUsNTIuMUwxMjUsNTIuMUwxMjUsNTIuMUwxMjUsNTIuMQ0KCQlMMTI1LDUyLjF6IE0xNDIsMTM1LjRIMTA4bDE3LTQwLjlMMTQyLDEzNS40eiIvPg0KPC9nPg0KPC9zdmc+DQo=" width="300">\n</div>\n<h2 _ngcontent-c0="">Here is the demo for Angular 4 and Polymer 2.0 web component two way data binding: </h2>\n\n<hr _ngcontent-c0="">\n\n\n<app-poly _ngcontent-c0="" _nghost-c1="">\n\n\n\n\n\n</app-poly>\n</app-root>\n<!-- <button onClick="adding()">adding event listener</button> -->\n\n\n</body></html>' }

Several different potential use cases revolving around screenshots:

• Generating screenshots endpoint. Maybe it allows you to specify some viewport options, so you can get a good size view
• An option to have screenshots generated and injected into response as the open graph/image sharing tags.

<meta property="og:image" content="generated-screenshot.png" />

With the current settings in the app.yaml config, it looks like the cost for App Engine on this project would be a minimum of $452/month. If I am a user looking to solve SEO for my blog that is hosted for almost nothing on Firebase, I could imagine there being quite a shock when all of my GCP free credits are used up in two thirds of a month.

1. Are the App Engine resource definitions a strong recommendation from the maintainers?
2. (If my math is correct) Could a warning be added in the README to caution casual projects from blindly deploying this and forgetting about it?

The math (https://cloud.google.com/appengine/pricing#flexible-environment-instances)

Assumptions

• App engine flex does not scale to 0.
• I still remember how to math.
• The pricing in the above link is correct.

Avg hours in a month = 730
vCPU = $0.0526 / core / hour
Mem = $0.0071 / GB / hour

8 vCPU and 28 GB of memory

$0.0526 * 730 * 8 = $307.184
$0.0071 * 730 * 28 = $145.124
$307.184 + $145.124 = ~$452 / month

Hello

I was doing load testing with rendertron.

I made about 200 requests to the same page.

about 17 of the requests got the "10 second time budget limit reached." error. and others got 304.

all the 17 responses had a status code of 200. and a body with everything except all the content in the shadowdom (like while all the content inside are missing)

as a result, the bot will see an empty page.

another experiment was when I tried to render yahoo.com

it also had this "10 second time budget limit reached." error.

but it can actually send back quite a few of the content.

I'd like to know how does this 10 second time budget limit reached. actually work.

The current implementation fully disables all cache used, which makes the server side rendering slower than it could be. (ref)

The browsers cache can be utilised for faster page renders and cache invalidation would work exactly the same way other browsers would need to handle caching:

• Normally you can't flush the browser's cache
• Normally you can't disable the browser's cache
• Normally you can't disable service workers

How cache invalidation would work:

• If a service worker is installed it will automatically check on page-load if there is a new version available and will automatically fetch all the latest files.
• If it is a classic application, the application would implement cache-headers with additional asset-signing and the application would handle it's self.

I was trying to deploy my own rendertron docker container to heroku.

1. the docker image was successfully built

2. heroku container:push web was successful as well

3. but when I visit the site. It gives an error. this is what i saw in the heroku logs

2017-08-30T20:03:01.757158+00:00 app[web.1]: npm ERR! 
2017-08-30T20:03:01.756957+00:00 app[web.1]: npm ERR! Exit status 1
2017-08-30T20:03:01.757338+00:00 app[web.1]: npm ERR! Failed at the [email protected] start script.
2017-08-30T20:03:01.757499+00:00 app[web.1]: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
2017-08-30T20:03:01.759830+00:00 app[web.1]: 
2017-08-30T20:03:01.760064+00:00 app[web.1]: npm ERR! A complete log of this run can be found in:
2017-08-30T20:03:01.760208+00:00 app[web.1]: npm ERR!     /app/.npm/_logs/2017-08-30T20_03_01_736Z-debug.log
2017-08-30T20:03:01.879956+00:00 heroku[web.1]: State changed from starting to crashed
2017-08-30T20:03:01.863596+00:00 heroku[web.1]: Process exited with status 1
2017-08-30T21:34:34.105177+00:00 heroku[web.1]: State changed from crashed to starting
2017-08-30T21:34:38.107376+00:00 heroku[web.1]: Starting process with command `npm start`
2017-08-30T21:34:41.628993+00:00 app[web.1]: 
2017-08-30T21:34:41.629007+00:00 app[web.1]: > [email protected] start /app
2017-08-30T21:34:41.629008+00:00 app[web.1]: > node src/main.js
2017-08-30T21:34:41.629009+00:00 app[web.1]: 
2017-08-30T21:34:43.153366+00:00 app[web.1]: Error: The environment variable CHROME_PATH must be set to executable of a build of Chromium version 54.0 or later.
2017-08-30T21:34:43.153377+00:00 app[web.1]:     at Object.linux (/app/node_modules/chrome-launcher/chrome-finder.js:105:15)
2017-08-30T21:34:43.153378+00:00 app[web.1]:     at Launcher.<anonymous> (/app/node_modules/chrome-launcher/chrome-launcher.js:109:75)
2017-08-30T21:34:43.153379+00:00 app[web.1]:     at Generator.next (<anonymous>)
2017-08-30T21:34:43.153380+00:00 app[web.1]:     at /app/node_modules/chrome-launcher/chrome-launcher.js:12:71
2017-08-30T21:34:43.153381+00:00 app[web.1]:     at Promise (<anonymous>)
2017-08-30T21:34:43.153381+00:00 app[web.1]:     at __awaiter (/app/node_modules/chrome-launcher/chrome-launcher.js:8:12)
2017-08-30T21:34:43.153382+00:00 app[web.1]:     at Launcher.launch (/app/node_modules/chrome-launcher/chrome-launcher.js:94:16)
2017-08-30T21:34:43.153383+00:00 app[web.1]:     at Object.<anonymous> (/app/node_modules/chrome-launcher/chrome-launcher.js:42:24)
2017-08-30T21:34:43.153383+00:00 app[web.1]:     at Generator.next (<anonymous>)
2017-08-30T21:34:43.153384+00:00 app[web.1]:     at /app/node_modules/chrome-launcher/chrome-launcher.js:12:71
2017-08-30T21:34:43.181613+00:00 app[web.1]: npm ERR! errno 1
2017-08-30T21:34:43.181255+00:00 app[web.1]: npm ERR! code ELIFECYCLE
2017-08-30T21:34:43.181894+00:00 app[web.1]: npm ERR! [email protected] start: `node src/main.js`
2017-08-30T21:34:43.182157+00:00 app[web.1]: npm ERR! Exit status 1
2017-08-30T21:34:43.182409+00:00 app[web.1]: npm ERR! 
2017-08-30T21:34:43.182705+00:00 app[web.1]: npm ERR! Failed at the [email protected] start script.
2017-08-30T21:34:43.182999+00:00 app[web.1]: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
2017-08-30T21:34:43.185002+00:00 app[web.1]: 
2017-08-30T21:34:43.185292+00:00 app[web.1]: npm ERR! A complete log of this run can be found in:
2017-08-30T21:34:43.185480+00:00 app[web.1]: npm ERR!     /app/.npm/_logs/2017-08-30T21_34_43_173Z-debug.log
2017-08-30T21:34:43.290883+00:00 heroku[web.1]: Process exited with status 1
2017-08-30T21:34:43.306186+00:00 heroku[web.1]: State changed from starting to crashed
2017-08-30T22:09:09.923840+00:00 app[api]: Deployed web (fa4a1d371d84) by user [email protected]
2017-08-30T22:09:09.923840+00:00 app[api]: Release v4 created by user [email protected]
2017-08-30T22:09:10.333913+00:00 heroku[web.1]: State changed from crashed to starting
2017-08-30T22:09:29.900213+00:00 heroku[web.1]: Starting process with command `run start`
2017-08-30T22:09:34.486108+00:00 heroku[web.1]: Process exited with status 1
2017-08-30T22:09:34.359622+00:00 app[web.1]: 
2017-08-30T22:09:34.359638+00:00 app[web.1]: Usage: npm <command>
2017-08-30T22:09:34.359639+00:00 app[web.1]: 
2017-08-30T22:09:34.359640+00:00 app[web.1]: where <command> is one of:
2017-08-30T22:09:34.359641+00:00 app[web.1]:     access, adduser, bin, bugs, c, cache, completion, config,
2017-08-30T22:09:34.359642+00:00 app[web.1]:     ddp, dedupe, deprecate, dist-tag, docs, doctor, edit,
2017-08-30T22:09:34.359643+00:00 app[web.1]:     explore, get, help, help-search, i, init, install,
2017-08-30T22:09:34.359643+00:00 app[web.1]:     install-test, it, link, list, ln, login, logout, ls,
2017-08-30T22:09:34.359644+00:00 app[web.1]:     outdated, owner, pack, ping, prefix, prune, publish, rb,
2017-08-30T22:09:34.359644+00:00 app[web.1]:     rebuild, repo, restart, root, run, run-script, s, se,
2017-08-30T22:09:34.359645+00:00 app[web.1]:     search, set, shrinkwrap, star, stars, start, stop, t, team,
2017-08-30T22:09:34.359646+00:00 app[web.1]:     version, view, whoami
2017-08-30T22:09:34.359647+00:00 app[web.1]: 
2017-08-30T22:09:34.359648+00:00 app[web.1]: npm <command> -h     quick help on <command>
2017-08-30T22:09:34.359646+00:00 app[web.1]:     test, tst, un, uninstall, unpublish, unstar, up, update, v,
2017-08-30T22:09:34.359648+00:00 app[web.1]: npm -l           display full usage info
2017-08-30T22:09:34.359649+00:00 app[web.1]: npm help <term>  search for help on <term>
2017-08-30T22:09:34.359650+00:00 app[web.1]: npm help npm     involved overview
2017-08-30T22:09:34.359650+00:00 app[web.1]: 
2017-08-30T22:09:34.359651+00:00 app[web.1]: Specify configs in the ini-formatted file:
2017-08-30T22:09:34.359651+00:00 app[web.1]:     /app/.npmrc
2017-08-30T22:09:34.359652+00:00 app[web.1]: or on the command line via: npm <command> --key value
2017-08-30T22:09:34.359653+00:00 app[web.1]: Config info can be viewed via: npm help config
2017-08-30T22:09:34.359653+00:00 app[web.1]: 
2017-08-30T22:09:34.359654+00:00 app[web.1]: [email protected] /nodejs/lib/node_modules/npm
2017-08-30T22:09:34.498472+00:00 heroku[web.1]: State changed from starting to crashed
2017-08-30T22:09:34.501309+00:00 heroku[web.1]: State changed from crashed to starting
2017-08-30T22:09:57.722801+00:00 heroku[web.1]: Starting process with command `run start`
2017-08-30T22:10:03.257312+00:00 heroku[web.1]: Process exited with status 1
2017-08-30T22:10:03.148639+00:00 app[web.1]: 
2017-08-30T22:10:03.148651+00:00 app[web.1]: Usage: npm <command>
2017-08-30T22:10:03.148652+00:00 app[web.1]: 
2017-08-30T22:10:03.148652+00:00 app[web.1]: where <command> is one of:
2017-08-30T22:10:03.148653+00:00 app[web.1]:     access, adduser, bin, bugs, c, cache, completion, config,
2017-08-30T22:10:03.148654+00:00 app[web.1]:     ddp, dedupe, deprecate, dist-tag, docs, doctor, edit,
2017-08-30T22:10:03.148655+00:00 app[web.1]:     explore, get, help, help-search, i, init, install,
2017-08-30T22:10:03.148655+00:00 app[web.1]:     install-test, it, link, list, ln, login, logout, ls,
2017-08-30T22:10:03.148656+00:00 app[web.1]:     outdated, owner, pack, ping, prefix, prune, publish, rb,
2017-08-30T22:10:03.148656+00:00 app[web.1]:     rebuild, repo, restart, root, run, run-script, s, se,
2017-08-30T22:10:03.148657+00:00 app[web.1]:     search, set, shrinkwrap, star, stars, start, stop, t, team,
2017-08-30T22:10:03.148658+00:00 app[web.1]:     test, tst, un, uninstall, unpublish, unstar, up, update, v,
2017-08-30T22:10:03.148658+00:00 app[web.1]:     version, view, whoami
2017-08-30T22:10:03.148659+00:00 app[web.1]: 
2017-08-30T22:10:03.148660+00:00 app[web.1]: npm <command> -h     quick help on <command>
2017-08-30T22:10:03.148660+00:00 app[web.1]: npm -l           display full usage info
2017-08-30T22:10:03.148661+00:00 app[web.1]: npm help <term>  search for help on <term>
2017-08-30T22:10:03.148661+00:00 app[web.1]: npm help npm     involved overview
2017-08-30T22:10:03.148662+00:00 app[web.1]: 
2017-08-30T22:10:03.148663+00:00 app[web.1]:     /app/.npmrc
2017-08-30T22:10:03.148662+00:00 app[web.1]: Specify configs in the ini-formatted file:
2017-08-30T22:10:03.148664+00:00 app[web.1]: or on the command line via: npm <command> --key value
2017-08-30T22:10:03.148665+00:00 app[web.1]: 
2017-08-30T22:10:03.148665+00:00 app[web.1]: Config info can be viewed via: npm help config
2017-08-30T22:10:03.148666+00:00 app[web.1]: [email protected] /nodejs/lib/node_modules/npm
2017-08-30T22:10:03.272913+00:00 heroku[web.1]: State changed from starting to crashed
2017-08-30T22:10:03.938038+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=infinite-forest-14908.herokuapp.com request_id=2523fa4f-cade-45fa-b20b-bcfcd9c44e16 fwd="204.128.192.32" dyno= connect= service= status=503 bytes= protocol=https
2017-08-30T22:10:05.126246+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=infinite-forest-14908.herokuapp.com request_id=4d265f87-bc97-416f-b71c-b678d8c635c3 fwd="204.128.192.32" dyno= connect= service= status=503 bytes= protocol=https
2017-08-30T22:10:06.284586+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=infinite-forest-14908.herokuapp.com request_id=9774d6aa-7144-40c4-8410-d26a0284f4a6 fwd="204.128.192.32" dyno= connect= service= status=503 bytes= protocol=https

I am a total newbie in terms of docker.

I'd like to know what went wrong.

thx!

https://github.com/GoogleChrome/puppeteer provides high level APIs for working with headless Chrome. Lot of rendertron's underlying code could be simplified (even removed) by adopting the puppeteer API.

Hello

Rendertron is using google-cloud/datastore for caching.

I'd like to know where these cached pages are storing.

Thank you!

Test out network interception on Chrome 61 to see if that would work.

Hello!

I was trying the screenshot demo with my website, and I noticed that it doesn't show emojis,
I've made a quick bin here: http://jsbin.com/qakokocihe

As you can see from the screenshot it is not rendering the Emojis and the Japanese Characters.

This is probably because it is missing the relative fonts.

Should not clear on every instance. Clearing also needs to go through every cache option.

Thanks for this awesome project. That's exactly what I've bee looking for. I just have problems getting this running with docker. I followed the steps in the readme to setup the docker container and used the same curl command to test the bot-render. I always got this output.

Seems like the express server can't connect to the chrome dev tools interface?!

{ Error: connect ECONNREFUSED 127.0.0.1:9222
    at Object.exports._errnoException (util.js:1026:11)
    at exports._exceptionWithHostPort (util.js:1049:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1136:14)
  code: 'ECONNREFUSED',
  errno: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 9222 }

Add an optional, in-memory rate limiting function.

Rate limit would be x pages rendered per hour. Notably, cache hits are excluded. If the limit is reached, return an appropriate status code & message indicating rate limit has been hit & maybe how long till the rate limit is reset or the next request will be processed.

@samuelli I'm going to start capturing some of the offline and ticket conversations around deployment to various providers beyond the GAE environment.

Would you prefer that this be in the readme or should I go about capturing this in the wiki or some other spot?

Hello

My ultimate goal is to use rendertron with PHP.

and I saw this in readme

Rendertron is also compatible with prerender.io middleware.

I have tried to use prerender middleware for expessJS.

app.use(require('prerender-node').set('prerenderServiceUrl', 'http://localhost:3000/render'));

but it just couldn't reach the rendertron end point.

I'd like to know if I can use prerender middleware for Zend Framework to get rendertron work with PHP. If so, what is needed to get this done?

also I'd like to know the minimum version of Node and Express required by rendertron. (in the readme is saying node 7+, but I read somewhere people are saying 7.60+ is needed.)

Thank you!

If Rendertron is started by npm start, the performance is awesome! It can render pretty much all the sites very fast.

If Rendertron is started as a docker container, it takes MUCH longer to do the rendering.

for example:

https://whispering-bayou-43373.herokuapp.com/list/mens_outerwear

docker container version of Rendertron takes 10+ secs

npm start version of Rendertron takes 4 secs

https://disneycruise.disney.go.com/

docker container version of Rendertron takes 8.6-10+ secs

npm start version of Rendertron takes 6-7 secs

Something needs to be done to the docker configuration.

Steps to reproduce

• run rendertron locally using npm start
• go to http://localhost:3000/
• paste the follow line into the input box
  https://whispering-bayou-43373.herokuapp.com/list/mens_outerwear
• click the render and serialize with web component v1

• build rendertron container docker build -t rendertron . --no-cache=true
• run rendertron container docker run -it -p 8080:8080 --cap-add SYS_ADMIN --name rendertron-container rendertron
• paste the following line into the input box
  https://whispering-bayou-43373.herokuapp.com/list/mens_outerwear
• click the render and serialize with web component v1

Results

npm start will give the rendering result in about 5 secs

docker run will give almost a blank page because it could not finish the rendering in 10 secs

My guess

since the poor performance is due to the long rendering time, my guess is that some of the docker configuration is making headless chrome run slow.

Updates

last week, I got the testing result as what I stated above after many many tests.

but this week, everything changed as if this is no longer the world I have lived in last week.

1. docker version of rendertron can render pretty much as fast as npm start version for AngularJS apps (GOOD!)

2. npm start version of rendertron is acting like the docker version of rendertron, and can no longer render several of the apps involve Polymer elements (BAD!)

https://whispering-bayou-43373.herokuapp.com/list/ladies_outerwear

https://mighty-dawn-10467.herokuapp.com/heroes

https://shop.polymer-project.org/list/ladies_outerwear

Allow for the cache to be invalidated somehow.

after watching the polymer summit video I decided to give rendertron a try

1. I run a local rendertron at localhost: 3000

2. I run a simple expressJS + polymer 2.0 app at localhost:1337

3. I enter localhost:1337 in the input field in the rendertron UI. it worked very well. it flatten shadow dom. and render the page correctly except the words in

   tags which it just ignored.

4. then I added the middleware below

app.use(rendertron.makeMiddleware({
  proxyUrl: 'localhost:3000/render/',
  injectShadyDom: true,
}));

to sever.js

5. the middleware can actually detect if a bot is making a request which is good

6. but it also gives me this error when a bot is making a request (yes I modified the middleware a bit to see what might go wrong)
   [rendertron middleware] undefined error fetching localhost:3000/render/http://localhost:1337/?wc-inject-shadydom=true

7. I simply copy and pasty
   localhost:3000/render/http://localhost:1337/?wc-inject-shadydom=true
   to a browser. and I can actually see a rendered version of the html by rendertron.

I'd like to know what actually went wrong and threw that error. and how to fix it.

thx!

Hello !

I'm really excited about rendertron since his presentation at the polymer summit and I have decided to try it with a personnal project.

But I got a problem with Polymer 2 and Polymerfire to retrieve data from Firebase Database

Description

Rendertron doesn't work with Polymerfire (https://github.com/firebase/polymerfire)

Expected outcome

Rendertron render the page and allow websocket to retrieve data

Actual outcome

No data is transfered through websocket with polymerfire (firebase-query or firebase-document)

Live Demo

I have set up a github repo with minimal polymer app and polymerfire (and a firebase database with products)

Repo : https://github.com/Oupsla/rendertron-polymerfire
Demo : https://rendertron-polymer.firebaseapp.com

Steps to reproduce

1. Init Polymer project
2. Add polymerfire (and in the app import polymerfire/firebase-app.html and polymerfire/firebase-query.html)
3. Retrieve collection of objects from a Firebase Database
4. Polymer serve this app (or deploy it)
5. Rendertron this app

Hi

When I tried to use rendertron and puppeteer to render the Angular 4 sample app Tours of Heroes, the angular 4 ngFor directives are ignored. All the content related to ngFor failed to render.

Due to the async request for heroes, Tour of Heroes actually has 2 stages of loading

1. load everything except for the ngFor stuff
2. load the ngFor stuff

It seems that rendertron thinks that after stage 1 of loading, the page has already finished loading. that's why it only renders the stuff without those ngFor async request content.

Since Sam was saying that rendertron can handle all those async requests during Polymer Summit, a fix would be nice~

thx!

When trying out https://www.product.cafe on render-tron.appspot.com I get the following error:

Failed to execute 'postMessage' on 'DOMWindow': The target origin provided ('https://www.product.cafe') does not match the recipient window's origin ('http://render-tron.appspot.com').
ha @ 0sTQzbapM8j.js?version=42:53

You can see the error here: http://render-tron.appspot.com/render/https://www.product.cafe?wc-inject-shadydom=true

Maybe also link rel="import"?

Under consideration: but allowing HTTP headers to be set via meta tags and/or JS API.

I was doing load testing on rendertron.

1. I make 100 request. the first one response is 200, and the rest is 304 which is as expected.

2. 5 mins later, I updated my app and deployed again. then I made another 10 request. this time the first response was 200, and the rest is 304.

based on the comment, cache is based on URL.

How come Rendertron actually knew the URL was updated and rendered it again instead of using the cache since it's within 24 hours visiting the same URL?

I'm struggling with running rendertron on AWS beanstalk as second container to my app. It is not possible to start the container in Dockerrun.aws.json version 2 due to security options not possible to set. Is it possible to bypass that?

Example container setup

{
  "AWSEBDockerrunVersion": 2,
  "volumes": [],
  "containerDefinitions": [
    {
      (my app here)
    },
    {
      "name": "rendertron",
      "image": "ID.dkr.ecr.eu-central-1.amazonaws.com/rendertron:latest",
      "essential": true,
      "memory": 256,
      "links": [
        "static"
      ],
      "dockerSecurityOptions": [
        "label:seccomp:/root/chrome.json"
      ]
    }
  ]
}

Hi

Rendertron can cache polymer 2.0 apps perfectly fine. But It failed to cached AngularJS or Angular 2/4 pages.

This happens to both npm run and running the --no-cache=false docker container.

Reproduction Steps

1. modify /rendertron/src/main.js by adding the following line below line 107 (const result = await renderer.serialize(request.params.url, request.query, config);)

console.log(result.status);

2. run rendertron using 'npm start'

3. try to render this site(a polymer 2.0 app) twice

https://enigmatic-beach-27289.herokuapp.com/

it will log 200 then 304

4. try to render this site(an AngularJS app) twice

https://disneycruise.disney.go.com/cruises-destinations/overview/bahamas/

it will log 200 then 200

Update

Three types of scenario for caching

1. Never caches. The response code from Rendertron always gives 200

samples (the second is AngularJS app):

https://www.google.com/

https://disneycruise.disney.go.com/cruises-destinations/overview/bermuda/

2. The cache is working fine. Rendertron always gives 304 after the first 200

samples: (the first one is AngularJS app. the second one is Polymer 2.0 App)

https://github.com/gothinkster/angularjs-realworld-example-app

https://boiling-tor-88178.herokuapp.com/7

for the github repo

• clone the repo

• cd into the folder

• npm install

• open src/js/config/app.config.js

• comment in line 12 $locationProvider.html5Mode(true);

• open src/index.html

• add <base href="/"> to the head

• run gulp

• go to localhost:4000

3. Cache works sometimes?

https://www.theguardian.com/us

https://www.theguardian.com/us/culture

The status code from Rendertron is totally random

It can be something like

200
200
200
304
304
304
200
200
304
200

Allow the cache to be warmed up. This could be in the form of providing site-maps which will then be queued and processed.

I can't play with it :

$ docker run -it -p 8080:8080 samdotli/rendertron

> [email protected] start /app
> node src/main.js

{ Error: connect ECONNREFUSED 127.0.0.1:44240
    at Object._errnoException (util.js:1041:11)
    at _exceptionWithHostPort (util.js:1064:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1153:14)
  code: 'ECONNREFUSED',
  errno: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 44240 }
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: `node src/main.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/botrender/.npm/_logs/2017-09-05T22_48_53_496Z-debug.log

I got ECONNREFUSED error running from npm start despite Chrome is running.

Uncaught exception
{ Error: connect ECONNREFUSED 127.0.0.1:34876
    at Object._errnoException (util.js:1041:11)
    at _exceptionWithHostPort (util.js:1064:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1153:14)
  code: 'ECONNREFUSED',
  errno: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 34876 }

I tried this one on my local Windows machine (it was fine), and then tried to run it on my DigitalOcean machine, but every time my express server redirects request to the Rendertron (or I request rendering from its UI) I get the error above.
Running Chrome version is:

google-chrome -version
Google Chrome 61.0.3163.100

The machine is running Ubuntu 16.04.
Maybe some setup hacks for just plain installation? Not dockerized one? Cause there are some instructions in README but for Docker image only, how to apply same fixes for non-dockerized version? Docker is heavy and my tiny machine will not survive its pressure.

Please help :) I really need this stuff :)
BTW, for some reason If I try to redirect renders to the https://render-tron.appspot.com/ instead of my local one (since I couldn't get it up and running) it doesn't render anything saying that it's unable to render URL :( Is it possible to use this endpoint from other IPs or whatever? :) Cause I have very low demand on site, so this even non-stable instance will be enough for me.

It appears that if you load a page that has more content than the viewport, the rest of the content does not get rendered. Are there any plans for full page screenshot support? Am I missing something in the docs?

Thanks.

Cache needs to also respect options:
eg. request route, it caches
request route with shady dom

Expected: get new render with shady dom
Actual: cached result w/o shady dom

Hi

I'm trying to wrap my mind around the significance of this project, and I'd like to get your feedback on my thinking about it, in case I'm missing something.

EDITED:

I have a React app that I wish to expose to search bots (for SEO.) I don’t believe in using SSR because it negates the scalability we get with Single Page Apps (SPAs) where we do all the CPU-bound work of rendering the page on the client as opposed to doing it on a shared server, where the most common scenario involves NodeJS as the server, a single-threaded, cooperative multitasking environment that is designed for I/O-bound work.

One of the alternatives I‘ve considered is as follows:

Upon receiving a request for a given URL, the server loads the index.html file into a string and inject into it all JSON and text data (as JS var declarations) for the initial requested route (which is specified in the hash fragment of the URL, including any query params), text data (into off-screen HTML intended for the bot) and the corresponding HTML meta tags. This way when the SPA is loaded by the browser the initial route will already have the data it needs, and will start rendering without having to wait for the data, so the initial render will feel fast since the browser won’t open a blank page then wait then render. Instead, it will open a blank page then immediately render, so the user won’t even see the blank page. In addition, the search bots will have the text content (in the off-screen div) and the meta tags to go along with the content. After the initial requested route is served, the SPA takes over.

The other alternative is to detect the user-agent and if it’s a bot then queue the request to a pool of Headless Chrome instances that will load the SPA like any user browser would, run it and return the HTML. The advantage is that my React app setup remains super simple and does not carry any server-side-rendering baggage.

Does my thinking on this make sense to you? What am I missing?

Thanks very much for sharing.