googlechrome / rendertron Goto Github PK
View Code? Open in Web Editor NEWA Headless Chrome rendering solution
Home Page: https://render-tron.appspot.com/
License: Apache License 2.0
A Headless Chrome rendering solution
Home Page: https://render-tron.appspot.com/
License: Apache License 2.0
http://localhost:3000/?url=https://polymer-shop2.appspot.com&wc-inject-shadydom=true
Maybe something to do with the styles?
I was trying to use rendertron to render an Angular 4 app which is using polymer 2.0 web components. but rendertron simply ignores the shadow doms. (shadydom is set to true of course.)
the rendering result still has the web component tags like
but inside the tags, it's total empty.
this is the rendering result (I modified rendertron to console log the rendering result)
body: '<html lang="en"><head><!-- Shady DOM styles for dom-bind --><!-- Shady DOM styles for dom-repeat --><!-- Shady DOM styles for dom-if --><!-- Shady DOM styles for array-selector --><!-- Shady DOM styles for custom-style --><!-- Shady DOM styles for paper-ripple --><!-- Shady DOM styles for iron-overlay-backdrop --><!-- Shady DOM styles for iron-meta --><!-- Shady DOM styles for iron-dropdown --><!-- Shady DOM styles for fade-in-animation --><!-- Shady DOM styles for fade-out-animation --><!-- Shady DOM styles for paper-menu-grow-height-animation --><!-- Shady DOM styles for paper-menu-grow-width-animation --><!-- Shady DOM styles for paper-menu-shrink-width-animation --><!-- Shady DOM styles for paper-menu-shrink-height-animation --><!-- Shady DOM styles for paper-menu-button --><!-- Shady DOM styles for paper-radio-button --><!-- Shady DOM styles for iron-a11y-announcer --><!-- Shady DOM styles for iron-input --><!-- Shady DOM styles for paper-input-char-counter --><!-- Shady DOM styles for paper-input-container --><!-- Shady DOM styles for paper-input-error --><!-- Shady DOM styles for paper-input --><!-- Shady DOM styles for paper-button --><!-- Shady DOM styles for my-carousel -->\n <meta charset="utf-8">\n <title>Demo</title>\n <base href="/">\n\n\t\n\t\n\t\n\t\n\t\n\n\t\n\n\t<!-- <dom-module id="x-custom">\n\t <template>\n\t <button on-click="handleClick">Kick Me</button>\n\t </template>\n\t <script>\n\n\t //console.log(this);\n\n\t //this.mediator = require(\'assets/mediator/mediator.min.js\').Mediator;\n\t //this.theMediator = new Mediator();\n\n\t class XCustom extends Polymer.Element {\n\n\t static get is() {return \'x-custom\'}\n\n\t handleClick() {\n\t console.log(\'Ow!\');\n\t var event = new Event(\'build\');\n\n\t }\n\t }\n\t customElements.define(XCustom.is, XCustom);\n\t </script>\n\t</dom-module> -->\n\n <meta name="viewport" content="width=device-width, initial-scale=1">\n <link rel="icon" type="image/x-icon" href="favicon.ico">\n \n \n\n<custom-style><style type="text/css" is="custom-style">/* You can add global styles to this file, and also import other style files */\n</style></custom-style><custom-style><style is="custom-style"></style></custom-style><base href="http://localhost:4200/"></head>\n<body>\n <app-root _nghost-c0="">\n<div _ngcontent-c0="" style="text-align:center">\n <h1 _ngcontent-c0="">\n Welcome to </h1>\n <img _ngcontent-c0="" src="data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4NCjwhLS0gR2VuZXJhdG9yOiBBZG9iZSBJbGx1c3RyYXRvciAxOS4xLjAsIFNWRyBFeHBvcnQgUGx1Zy1JbiAuIFNWRyBWZXJzaW9uOiA2LjAwIEJ1aWxkIDApICAtLT4NCjxzdmcgdmVyc2lvbj0iMS4xIiBpZD0iTGF5ZXJfMSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayIgeD0iMHB4IiB5PSIwcHgiDQoJIHZpZXdCb3g9IjAgMCAyNTAgMjUwIiBzdHlsZT0iZW5hYmxlLWJhY2tncm91bmQ6bmV3IDAgMCAyNTAgMjUwOyIgeG1sOnNwYWNlPSJwcmVzZXJ2ZSI+DQo8c3R5bGUgdHlwZT0idGV4dC9jc3MiPg0KCS5zdDB7ZmlsbDojREQwMDMxO30NCgkuc3Qxe2ZpbGw6I0MzMDAyRjt9DQoJLnN0MntmaWxsOiNGRkZGRkY7fQ0KPC9zdHlsZT4NCjxnPg0KCTxwb2x5Z29uIGNsYXNzPSJzdDAiIHBvaW50cz0iMTI1LDMwIDEyNSwzMCAxMjUsMzAgMzEuOSw2My4yIDQ2LjEsMTg2LjMgMTI1LDIzMCAxMjUsMjMwIDEyNSwyMzAgMjAzLjksMTg2LjMgMjE4LjEsNjMuMiAJIi8+DQoJPHBvbHlnb24gY2xhc3M9InN0MSIgcG9pbnRzPSIxMjUsMzAgMTI1LDUyLjIgMTI1LDUyLjEgMTI1LDE1My40IDEyNSwxNTMuNCAxMjUsMjMwIDEyNSwyMzAgMjAzLjksMTg2LjMgMjE4LjEsNjMuMiAxMjUsMzAgCSIvPg0KCTxwYXRoIGNsYXNzPSJzdDIiIGQ9Ik0xMjUsNTIuMUw2Ni44LDE4Mi42aDBoMjEuN2gwbDExLjctMjkuMmg0OS40bDExLjcsMjkuMmgwaDIxLjdoMEwxMjUsNTIuMUwxMjUsNTIuMUwxMjUsNTIuMUwxMjUsNTIuMQ0KCQlMMTI1LDUyLjF6IE0xNDIsMTM1LjRIMTA4bDE3LTQwLjlMMTQyLDEzNS40eiIvPg0KPC9nPg0KPC9zdmc+DQo=" width="300">\n</div>\n<h2 _ngcontent-c0="">Here is the demo for Angular 4 and Polymer 2.0 web component two way data binding: </h2>\n\n<hr _ngcontent-c0="">\n\n\n<app-poly _ngcontent-c0="" _nghost-c1="">\n\n\n\n\n\n</app-poly>\n</app-root>\n<!-- <button onClick="adding()">adding event listener</button> -->\n\n\n</body></html>' }
I was trying to use rendertron for the polymer sample shopping app.
when I test it locally it works fine and fast.
after I deploy, and try to visit it as a bot, I got this error
10 second time budget limit reached.
Attempted rendering: http://whispering-bayou-43373.herokuapp.com/list/mens_outerwear
Page load event fired: false
I'd like to know what makes it fail to detect the page load event.
PS: if you visit the page as a regular user everything is fine.
PPS: I removed all the pictures on purpose.
Under consideration: Allow for a flag to notify when rendering is complete.
Need to find cases where the current loading function is insufficient. Do you have an example? Please comment below!
Hello!
I was trying the screenshot demo with my website, and I noticed that it doesn't show emojis,
I've made a quick bin here: http://jsbin.com/qakokocihe
As you can see from the screenshot it is not rendering the Emojis and the Japanese Characters.
This is probably because it is missing the relative fonts.
I got ECONNREFUSED error running from npm start
despite Chrome is running.
Uncaught exception
{ Error: connect ECONNREFUSED 127.0.0.1:34876
at Object._errnoException (util.js:1041:11)
at _exceptionWithHostPort (util.js:1064:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1153:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 34876 }
I tried this one on my local Windows machine (it was fine), and then tried to run it on my DigitalOcean machine, but every time my express server redirects request to the Rendertron (or I request rendering from its UI) I get the error above.
Running Chrome version is:
google-chrome -version
Google Chrome 61.0.3163.100
The machine is running Ubuntu 16.04.
Maybe some setup hacks for just plain installation? Not dockerized one? Cause there are some instructions in README but for Docker image only, how to apply same fixes for non-dockerized version? Docker is heavy and my tiny machine will not survive its pressure.
Please help :) I really need this stuff :)
BTW, for some reason If I try to redirect renders to the https://render-tron.appspot.com/ instead of my local one (since I couldn't get it up and running) it doesn't render anything saying that it's unable to render URL :( Is it possible to use this endpoint from other IPs or whatever? :) Cause I have very low demand on site, so this even non-stable instance will be enough for me.
When trying out https://www.product.cafe on render-tron.appspot.com I get the following error:
Failed to execute 'postMessage' on 'DOMWindow': The target origin provided ('https://www.product.cafe') does not match the recipient window's origin ('http://render-tron.appspot.com').
ha @ 0sTQzbapM8j.js?version=42:53
You can see the error here: http://render-tron.appspot.com/render/https://www.product.cafe?wc-inject-shadydom=true
Allow the cache to be warmed up. This could be in the form of providing site-maps which will then be queued and processed.
I can't play with it :
$ docker run -it -p 8080:8080 samdotli/rendertron
> [email protected] start /app
> node src/main.js
{ Error: connect ECONNREFUSED 127.0.0.1:44240
at Object._errnoException (util.js:1041:11)
at _exceptionWithHostPort (util.js:1064:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1153:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 44240 }
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: `node src/main.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /home/botrender/.npm/_logs/2017-09-05T22_48_53_496Z-debug.log
Test out network interception on Chrome 61 to see if that would work.
I was doing load testing on rendertron.
I make 100 request. the first one response is 200, and the rest is 304 which is as expected.
5 mins later, I updated my app and deployed again. then I made another 10 request. this time the first response was 200, and the rest is 304.
based on the comment, cache is based on URL.
How come Rendertron actually knew the URL was updated and rendered it again instead of using the cache since it's within 24 hours visiting the same URL?
Hi
I'm trying to wrap my mind around the significance of this project, and I'd like to get your feedback on my thinking about it, in case I'm missing something.
EDITED:
I have a React app that I wish to expose to search bots (for SEO.) I don’t believe in using SSR because it negates the scalability we get with Single Page Apps (SPAs) where we do all the CPU-bound work of rendering the page on the client as opposed to doing it on a shared server, where the most common scenario involves NodeJS as the server, a single-threaded, cooperative multitasking environment that is designed for I/O-bound work.
One of the alternatives I‘ve considered is as follows:
Upon receiving a request for a given URL, the server loads the index.html file into a string and inject into it all JSON and text data (as JS var declarations) for the initial requested route (which is specified in the hash fragment of the URL, including any query params), text data (into off-screen HTML intended for the bot) and the corresponding HTML meta tags. This way when the SPA is loaded by the browser the initial route will already have the data it needs, and will start rendering without having to wait for the data, so the initial render will feel fast since the browser won’t open a blank page then wait then render. Instead, it will open a blank page then immediately render, so the user won’t even see the blank page. In addition, the search bots will have the text content (in the off-screen div) and the meta tags to go along with the content. After the initial requested route is served, the SPA takes over.
The other alternative is to detect the user-agent and if it’s a bot then queue the request to a pool of Headless Chrome instances that will load the SPA like any user browser would, run it and return the HTML. The advantage is that my React app setup remains super simple and does not carry any server-side-rendering baggage.
Does my thinking on this make sense to you? What am I missing?
Thanks very much for sharing.
Thanks for this awesome project. That's exactly what I've bee looking for. I just have problems getting this running with docker. I followed the steps in the readme to setup the docker container and used the same curl command to test the bot-render. I always got this output.
Seems like the express server can't connect to the chrome dev tools interface?!
{ Error: connect ECONNREFUSED 127.0.0.1:9222
at Object.exports._errnoException (util.js:1026:11)
at exports._exceptionWithHostPort (util.js:1049:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1136:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 9222 }
Adding an address without protocol gives
Eg : google.com
Cannot render google.com - "First argument must be a string, Buffer, ArrayBuffer, Array, or array-like object."
Maybe support for building the URL from the address provided ?
Under consideration: but allowing HTTP headers to be set via meta tags and/or JS API.
I was trying to deploy my own rendertron docker container to heroku.
the docker image was successfully built
heroku container:push web
was successful as well
but when I visit the site. It gives an error. this is what i saw in the heroku logs
2017-08-30T20:03:01.757158+00:00 app[web.1]: npm ERR!
2017-08-30T20:03:01.756957+00:00 app[web.1]: npm ERR! Exit status 1
2017-08-30T20:03:01.757338+00:00 app[web.1]: npm ERR! Failed at the [email protected] start script.
2017-08-30T20:03:01.757499+00:00 app[web.1]: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
2017-08-30T20:03:01.759830+00:00 app[web.1]:
2017-08-30T20:03:01.760064+00:00 app[web.1]: npm ERR! A complete log of this run can be found in:
2017-08-30T20:03:01.760208+00:00 app[web.1]: npm ERR! /app/.npm/_logs/2017-08-30T20_03_01_736Z-debug.log
2017-08-30T20:03:01.879956+00:00 heroku[web.1]: State changed from starting to crashed
2017-08-30T20:03:01.863596+00:00 heroku[web.1]: Process exited with status 1
2017-08-30T21:34:34.105177+00:00 heroku[web.1]: State changed from crashed to starting
2017-08-30T21:34:38.107376+00:00 heroku[web.1]: Starting process with command `npm start`
2017-08-30T21:34:41.628993+00:00 app[web.1]:
2017-08-30T21:34:41.629007+00:00 app[web.1]: > [email protected] start /app
2017-08-30T21:34:41.629008+00:00 app[web.1]: > node src/main.js
2017-08-30T21:34:41.629009+00:00 app[web.1]:
2017-08-30T21:34:43.153366+00:00 app[web.1]: Error: The environment variable CHROME_PATH must be set to executable of a build of Chromium version 54.0 or later.
2017-08-30T21:34:43.153377+00:00 app[web.1]: at Object.linux (/app/node_modules/chrome-launcher/chrome-finder.js:105:15)
2017-08-30T21:34:43.153378+00:00 app[web.1]: at Launcher.<anonymous> (/app/node_modules/chrome-launcher/chrome-launcher.js:109:75)
2017-08-30T21:34:43.153379+00:00 app[web.1]: at Generator.next (<anonymous>)
2017-08-30T21:34:43.153380+00:00 app[web.1]: at /app/node_modules/chrome-launcher/chrome-launcher.js:12:71
2017-08-30T21:34:43.153381+00:00 app[web.1]: at Promise (<anonymous>)
2017-08-30T21:34:43.153381+00:00 app[web.1]: at __awaiter (/app/node_modules/chrome-launcher/chrome-launcher.js:8:12)
2017-08-30T21:34:43.153382+00:00 app[web.1]: at Launcher.launch (/app/node_modules/chrome-launcher/chrome-launcher.js:94:16)
2017-08-30T21:34:43.153383+00:00 app[web.1]: at Object.<anonymous> (/app/node_modules/chrome-launcher/chrome-launcher.js:42:24)
2017-08-30T21:34:43.153383+00:00 app[web.1]: at Generator.next (<anonymous>)
2017-08-30T21:34:43.153384+00:00 app[web.1]: at /app/node_modules/chrome-launcher/chrome-launcher.js:12:71
2017-08-30T21:34:43.181613+00:00 app[web.1]: npm ERR! errno 1
2017-08-30T21:34:43.181255+00:00 app[web.1]: npm ERR! code ELIFECYCLE
2017-08-30T21:34:43.181894+00:00 app[web.1]: npm ERR! [email protected] start: `node src/main.js`
2017-08-30T21:34:43.182157+00:00 app[web.1]: npm ERR! Exit status 1
2017-08-30T21:34:43.182409+00:00 app[web.1]: npm ERR!
2017-08-30T21:34:43.182705+00:00 app[web.1]: npm ERR! Failed at the [email protected] start script.
2017-08-30T21:34:43.182999+00:00 app[web.1]: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
2017-08-30T21:34:43.185002+00:00 app[web.1]:
2017-08-30T21:34:43.185292+00:00 app[web.1]: npm ERR! A complete log of this run can be found in:
2017-08-30T21:34:43.185480+00:00 app[web.1]: npm ERR! /app/.npm/_logs/2017-08-30T21_34_43_173Z-debug.log
2017-08-30T21:34:43.290883+00:00 heroku[web.1]: Process exited with status 1
2017-08-30T21:34:43.306186+00:00 heroku[web.1]: State changed from starting to crashed
2017-08-30T22:09:09.923840+00:00 app[api]: Deployed web (fa4a1d371d84) by user [email protected]
2017-08-30T22:09:09.923840+00:00 app[api]: Release v4 created by user [email protected]
2017-08-30T22:09:10.333913+00:00 heroku[web.1]: State changed from crashed to starting
2017-08-30T22:09:29.900213+00:00 heroku[web.1]: Starting process with command `run start`
2017-08-30T22:09:34.486108+00:00 heroku[web.1]: Process exited with status 1
2017-08-30T22:09:34.359622+00:00 app[web.1]:
2017-08-30T22:09:34.359638+00:00 app[web.1]: Usage: npm <command>
2017-08-30T22:09:34.359639+00:00 app[web.1]:
2017-08-30T22:09:34.359640+00:00 app[web.1]: where <command> is one of:
2017-08-30T22:09:34.359641+00:00 app[web.1]: access, adduser, bin, bugs, c, cache, completion, config,
2017-08-30T22:09:34.359642+00:00 app[web.1]: ddp, dedupe, deprecate, dist-tag, docs, doctor, edit,
2017-08-30T22:09:34.359643+00:00 app[web.1]: explore, get, help, help-search, i, init, install,
2017-08-30T22:09:34.359643+00:00 app[web.1]: install-test, it, link, list, ln, login, logout, ls,
2017-08-30T22:09:34.359644+00:00 app[web.1]: outdated, owner, pack, ping, prefix, prune, publish, rb,
2017-08-30T22:09:34.359644+00:00 app[web.1]: rebuild, repo, restart, root, run, run-script, s, se,
2017-08-30T22:09:34.359645+00:00 app[web.1]: search, set, shrinkwrap, star, stars, start, stop, t, team,
2017-08-30T22:09:34.359646+00:00 app[web.1]: version, view, whoami
2017-08-30T22:09:34.359647+00:00 app[web.1]:
2017-08-30T22:09:34.359648+00:00 app[web.1]: npm <command> -h quick help on <command>
2017-08-30T22:09:34.359646+00:00 app[web.1]: test, tst, un, uninstall, unpublish, unstar, up, update, v,
2017-08-30T22:09:34.359648+00:00 app[web.1]: npm -l display full usage info
2017-08-30T22:09:34.359649+00:00 app[web.1]: npm help <term> search for help on <term>
2017-08-30T22:09:34.359650+00:00 app[web.1]: npm help npm involved overview
2017-08-30T22:09:34.359650+00:00 app[web.1]:
2017-08-30T22:09:34.359651+00:00 app[web.1]: Specify configs in the ini-formatted file:
2017-08-30T22:09:34.359651+00:00 app[web.1]: /app/.npmrc
2017-08-30T22:09:34.359652+00:00 app[web.1]: or on the command line via: npm <command> --key value
2017-08-30T22:09:34.359653+00:00 app[web.1]: Config info can be viewed via: npm help config
2017-08-30T22:09:34.359653+00:00 app[web.1]:
2017-08-30T22:09:34.359654+00:00 app[web.1]: [email protected] /nodejs/lib/node_modules/npm
2017-08-30T22:09:34.498472+00:00 heroku[web.1]: State changed from starting to crashed
2017-08-30T22:09:34.501309+00:00 heroku[web.1]: State changed from crashed to starting
2017-08-30T22:09:57.722801+00:00 heroku[web.1]: Starting process with command `run start`
2017-08-30T22:10:03.257312+00:00 heroku[web.1]: Process exited with status 1
2017-08-30T22:10:03.148639+00:00 app[web.1]:
2017-08-30T22:10:03.148651+00:00 app[web.1]: Usage: npm <command>
2017-08-30T22:10:03.148652+00:00 app[web.1]:
2017-08-30T22:10:03.148652+00:00 app[web.1]: where <command> is one of:
2017-08-30T22:10:03.148653+00:00 app[web.1]: access, adduser, bin, bugs, c, cache, completion, config,
2017-08-30T22:10:03.148654+00:00 app[web.1]: ddp, dedupe, deprecate, dist-tag, docs, doctor, edit,
2017-08-30T22:10:03.148655+00:00 app[web.1]: explore, get, help, help-search, i, init, install,
2017-08-30T22:10:03.148655+00:00 app[web.1]: install-test, it, link, list, ln, login, logout, ls,
2017-08-30T22:10:03.148656+00:00 app[web.1]: outdated, owner, pack, ping, prefix, prune, publish, rb,
2017-08-30T22:10:03.148656+00:00 app[web.1]: rebuild, repo, restart, root, run, run-script, s, se,
2017-08-30T22:10:03.148657+00:00 app[web.1]: search, set, shrinkwrap, star, stars, start, stop, t, team,
2017-08-30T22:10:03.148658+00:00 app[web.1]: test, tst, un, uninstall, unpublish, unstar, up, update, v,
2017-08-30T22:10:03.148658+00:00 app[web.1]: version, view, whoami
2017-08-30T22:10:03.148659+00:00 app[web.1]:
2017-08-30T22:10:03.148660+00:00 app[web.1]: npm <command> -h quick help on <command>
2017-08-30T22:10:03.148660+00:00 app[web.1]: npm -l display full usage info
2017-08-30T22:10:03.148661+00:00 app[web.1]: npm help <term> search for help on <term>
2017-08-30T22:10:03.148661+00:00 app[web.1]: npm help npm involved overview
2017-08-30T22:10:03.148662+00:00 app[web.1]:
2017-08-30T22:10:03.148663+00:00 app[web.1]: /app/.npmrc
2017-08-30T22:10:03.148662+00:00 app[web.1]: Specify configs in the ini-formatted file:
2017-08-30T22:10:03.148664+00:00 app[web.1]: or on the command line via: npm <command> --key value
2017-08-30T22:10:03.148665+00:00 app[web.1]:
2017-08-30T22:10:03.148665+00:00 app[web.1]: Config info can be viewed via: npm help config
2017-08-30T22:10:03.148666+00:00 app[web.1]: [email protected] /nodejs/lib/node_modules/npm
2017-08-30T22:10:03.272913+00:00 heroku[web.1]: State changed from starting to crashed
2017-08-30T22:10:03.938038+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=infinite-forest-14908.herokuapp.com request_id=2523fa4f-cade-45fa-b20b-bcfcd9c44e16 fwd="204.128.192.32" dyno= connect= service= status=503 bytes= protocol=https
2017-08-30T22:10:05.126246+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=infinite-forest-14908.herokuapp.com request_id=4d265f87-bc97-416f-b71c-b678d8c635c3 fwd="204.128.192.32" dyno= connect= service= status=503 bytes= protocol=https
2017-08-30T22:10:06.284586+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=infinite-forest-14908.herokuapp.com request_id=9774d6aa-7144-40c4-8410-d26a0284f4a6 fwd="204.128.192.32" dyno= connect= service= status=503 bytes= protocol=https
I am a total newbie in terms of docker.
I'd like to know what went wrong.
thx!
Allow the status code to be set within the response. Meta tag to start:
<meta name="bot-render-status" content="404">
Perhaps a JS API:
x.setStatusCode(404)
Hello
Rendertron is using google-cloud/datastore for caching.
I'd like to know where these cached pages are storing.
Thank you!
Hello !
I'm really excited about rendertron since his presentation at the polymer summit and I have decided to try it with a personnal project.
But I got a problem with Polymer 2 and Polymerfire to retrieve data from Firebase Database
Rendertron doesn't work with Polymerfire (https://github.com/firebase/polymerfire)
Rendertron render the page and allow websocket to retrieve data
No data is transfered through websocket with polymerfire (firebase-query or firebase-document)
I have set up a github repo with minimal polymer app and polymerfire (and a firebase database with products)
Repo : https://github.com/Oupsla/rendertron-polymerfire
Demo : https://rendertron-polymer.firebaseapp.com
@samuelli I'm going to start capturing some of the offline and ticket conversations around deployment to various providers beyond the GAE environment.
Would you prefer that this be in the readme or should I go about capturing this in the wiki or some other spot?
Everyone is saying that googlebot has no problem clawing polymer 2.0 websites.
according to polymer-project.org, it seems like a true statement.
I deployed a very simple polymer 2.0 app on heroku using express to serve the index.html.
and try to use google console to fetch the page.
Sadly, the result didn't show anything inside the shadow dom.
and currently googlebot is not on the user agent list.
Maybe we should add googlebot to the user agent list for rendertron middleware as well?
Allow for the cache to be invalidated somehow.
Hi
Rendertron can cache polymer 2.0 apps perfectly fine. But It failed to cached AngularJS or Angular 2/4 pages.
This happens to both npm run and running the --no-cache=false docker container.
/rendertron/src/main.js
by adding the following line below line 107 (const result = await renderer.serialize(request.params.url, request.query, config);
)console.log(result.status);
run rendertron using 'npm start'
try to render this site(a polymer 2.0 app) twice
https://enigmatic-beach-27289.herokuapp.com/
it will log 200 then 304
https://disneycruise.disney.go.com/cruises-destinations/overview/bahamas/
it will log 200 then 200
samples (the second is AngularJS app):
https://www.google.com/
https://disneycruise.disney.go.com/cruises-destinations/overview/bermuda/
samples: (the first one is AngularJS app. the second one is Polymer 2.0 App)
https://github.com/gothinkster/angularjs-realworld-example-app
https://boiling-tor-88178.herokuapp.com/7
for the github repo
clone the repo
cd into the folder
npm install
open src/js/config/app.config.js
comment in line 12 $locationProvider.html5Mode(true);
open src/index.html
add <base href="/">
to the head
run gulp
go to localhost:4000
https://www.theguardian.com/us
https://www.theguardian.com/us/culture
The status code from Rendertron is totally random
It can be something like
200
200
200
304
304
304
200
200
304
200
By testing Rendertron, I came across a few sites rendertron is unable to render.
How about let's have a post for people to list the sites rendertron is not able to render?
I think it will make it easier for us to figure out what the pattern is and what needs to be fixed.
for example
https://mighty-dawn-10467.herokuapp.com/heroes
Angular4 + polymer 2.0 app based on the Angular 4 official tutorial Tour of Heroes
Note: I was able to use Rendertron to render this site before and let google/bing bots to successfully fetch the rendering result. It was perfect. But now Rendertron cannot even render it anymore.
https://whispering-bayou-43373.herokuapp.com/list/ladies_outerwear
Polymer 2.0 sample shopping app from Polymer CLI
Note: I was able to use
npm start
version of Rendertron to render this polymer 2.0 app. but now Rendertron cannot render it anymore.
Should not clear on every instance. Clearing also needs to go through every cache option.
Several different potential use cases revolving around screenshots:
<meta property="og:image" content="generated-screenshot.png" />
https://github.com/GoogleChrome/puppeteer provides high level APIs for working with headless Chrome. Lot of rendertron's underlying code could be simplified (even removed) by adopting the puppeteer API.
First of all, I really appreciate your excellent work!
We've been thinking building a similar screenshot service after Chrome Headless come out, glad we found rendertron before we start developing our own.
The thing is, we intend to use this screenshot/render service in production environment but couldn't find any benchmark result or pressure test result, like how many screenshot requests can it take parallelly?
Do you have any plan doing this, how can we help out?
Maybe also link rel="import"?
Hello
I was doing load testing with rendertron.
I made about 200 requests to the same page.
about 17 of the requests got the "10 second time budget limit reached." error. and others got 304.
all the 17 responses had a status code of 200. and a body with everything except all the content in the shadowdom (like while all the content inside are missing)
as a result, the bot will see an empty page.
another experiment was when I tried to render yahoo.com
it also had this "10 second time budget limit reached." error.
but it can actually send back quite a few of the content.
I'd like to know how does this 10 second time budget limit reached. actually work.
I have a notion that I'd like to add a cache layer that checks to see if the site has been rendered and stored in the last X amount of seconds. I have two ideas around this:
Option one obviously binds the container to be less portable (and I'd think this would be an enabled feature rather than a requirement to run the container). Option two has more flexibility (not bound to GCP), but still sits in the same boat (probably shouldn't be a requirement).
Would this be something that would be interesting to you @samuelli as a PR? Did you have any thoughts on the topic of caching?
Add an optional, in-memory rate limiting function.
Rate limit would be x pages rendered per hour. Notably, cache hits are excluded. If the limit is reached, return an appropriate status code & message indicating rate limit has been hit & maybe how long till the rate limit is reset or the next request will be processed.
Instant of everyone need to build the image, it could be useful if we could just pull the image from Docker Hub or some other Docker Registry (quay? Google Cloud Registry?)
ubuntu16.04 Docker version 17.03.1-ce, build c6d412e
When I tried
docker exec bot-render-container curl http://localhost:8080/?url=https://dynamic-meta.appspot.com
docker console output following logs
Listening on port 8080
{ Error: connect ECONNREFUSED 127.0.0.1:9222
at Object.exports._errnoException (util.js:1014:11)
at exports._exceptionWithHostPort (util.js:1037:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1138:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 9222 }
The current implementation fully disables all cache used, which makes the server side rendering slower than it could be. (ref)
The browsers cache can be utilised for faster page renders and cache invalidation would work exactly the same way other browsers would need to handle caching:
How cache invalidation would work:
If Rendertron is started by npm start, the performance is awesome! It can render pretty much all the sites very fast.
If Rendertron is started as a docker container, it takes MUCH longer to do the rendering.
for example:
https://whispering-bayou-43373.herokuapp.com/list/mens_outerwear
docker container version of Rendertron takes 10+ secs
npm start version of Rendertron takes 4 secs
https://disneycruise.disney.go.com/
docker container version of Rendertron takes 8.6-10+ secs
npm start version of Rendertron takes 6-7 secs
Something needs to be done to the docker configuration.
npm start
http://localhost:3000/
https://whispering-bayou-43373.herokuapp.com/list/mens_outerwear
docker build -t rendertron . --no-cache=true
docker run -it -p 8080:8080 --cap-add SYS_ADMIN --name rendertron-container rendertron
https://whispering-bayou-43373.herokuapp.com/list/mens_outerwear
npm start will give the rendering result in about 5 secs
docker run will give almost a blank page because it could not finish the rendering in 10 secs
since the poor performance is due to the long rendering time, my guess is that some of the docker configuration is making headless chrome run slow.
last week, I got the testing result as what I stated above after many many tests.
but this week, everything changed as if this is no longer the world I have lived in last week.
docker version of rendertron can render pretty much as fast as npm start version for AngularJS apps (GOOD!)
npm start version of rendertron is acting like the docker version of rendertron, and can no longer render several of the apps involve Polymer elements (BAD!)
https://whispering-bayou-43373.herokuapp.com/list/ladies_outerwear
https://mighty-dawn-10467.herokuapp.com/heroes
https://shop.polymer-project.org/list/ladies_outerwear
screenshot/https://fonts.google.com/?width=1280&height=1299 still hung after 85s. No errors, so something async never resolved.
after watching the polymer summit video I decided to give rendertron a try
I run a local rendertron at localhost: 3000
I run a simple expressJS + polymer 2.0 app at localhost:1337
I enter localhost:1337 in the input field in the rendertron UI. it worked very well. it flatten shadow dom. and render the page correctly except the words in
tags which it just ignored.
then I added the middleware below
app.use(rendertron.makeMiddleware({
proxyUrl: 'localhost:3000/render/',
injectShadyDom: true,
}));
to sever.js
the middleware can actually detect if a bot is making a request which is good
but it also gives me this error when a bot is making a request (yes I modified the middleware a bit to see what might go wrong)
[rendertron middleware] undefined error fetching localhost:3000/render/http://localhost:1337/?wc-inject-shadydom=true
I simply copy and pasty
localhost:3000/render/http://localhost:1337/?wc-inject-shadydom=true
to a browser. and I can actually see a rendered version of the html by rendertron.
I'd like to know what actually went wrong and threw that error. and how to fix it.
thx!
With the current settings in the app.yaml config, it looks like the cost for App Engine on this project would be a minimum of $452/month. If I am a user looking to solve SEO for my blog that is hosted for almost nothing on Firebase, I could imagine there being quite a shock when all of my GCP free credits are used up in two thirds of a month.
The math (https://cloud.google.com/appengine/pricing#flexible-environment-instances)
Assumptions
Avg hours in a month = 730
vCPU = $0.0526 / core / hour
Mem = $0.0071 / GB / hour
8 vCPU and 28 GB of memory
$0.0526 * 730 * 8 = $307.184
$0.0071 * 730 * 28 = $145.124
$307.184 + $145.124 = ~$452 / month
Hi
When I tried to use rendertron and puppeteer to render the Angular 4 sample app Tours of Heroes, the angular 4 ngFor directives are ignored. All the content related to ngFor failed to render.
Due to the async request for heroes, Tour of Heroes actually has 2 stages of loading
It seems that rendertron thinks that after stage 1 of loading, the page has already finished loading. that's why it only renders the stuff without those ngFor async request content.
Since Sam was saying that rendertron can handle all those async requests during Polymer Summit, a fix would be nice~
thx!
It appears that if you load a page that has more content than the viewport, the rest of the content does not get rendered. Are there any plans for full page screenshot support? Am I missing something in the docs?
Thanks.
Cache needs to also respect options:
eg. request route, it caches
request route with shady dom
Expected: get new render with shady dom
Actual: cached result w/o shady dom
I'm struggling with running rendertron on AWS beanstalk as second container to my app. It is not possible to start the container in Dockerrun.aws.json version 2 due to security options not possible to set. Is it possible to bypass that?
Example container setup
{
"AWSEBDockerrunVersion": 2,
"volumes": [],
"containerDefinitions": [
{
(my app here)
},
{
"name": "rendertron",
"image": "ID.dkr.ecr.eu-central-1.amazonaws.com/rendertron:latest",
"essential": true,
"memory": 256,
"links": [
"static"
],
"dockerSecurityOptions": [
"label:seccomp:/root/chrome.json"
]
}
]
}
Hello
My ultimate goal is to use rendertron with PHP.
and I saw this in readme
Rendertron is also compatible with prerender.io middleware.
I have tried to use prerender middleware for expessJS.
app.use(require('prerender-node').set('prerenderServiceUrl', 'http://localhost:3000/render'));
but it just couldn't reach the rendertron end point.
I'd like to know if I can use prerender middleware for Zend Framework to get rendertron work with PHP. If so, what is needed to get this done?
also I'd like to know the minimum version of Node and Express required by rendertron. (in the readme is saying node 7+, but I read somewhere people are saying 7.60+ is needed.)
Thank you!
Allow for a config file.
cache: {
enabled: true,
freshness: 1440
},
etc
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.