Currently hardcoded in dashboard.handlebars. See TODO at the bottom of that file.

Stop cloning from github in dockerfiles. Update package.json / bower.json to use the new version.

Filter out any messages sent from the frontend client to the kernel at the backend server except for ipywidgets comm messages on the shell channel. Allowing any other message gives the user the ability to execute arbitrary code. (In fact, it's not yet know if locking down to comm messages only is good enough to prevent code execution or not yet, but it's a step in the right direction.)

See https://github.com/jupyter-incubator/dashboards/wiki/Deployed-Dashboard-Threat-Analysis

Right now we're using a somewhat home grown recipe that handles text frames well, but doesn't do anything with ping or pong or any of the other WS protocol op codes. This is biting us at the moment because the client's pongs are not making it to the kernel gateway in response to its pings, and so the KG kills the websocket connections after 90 seconds thinking they're dead.

I'm going to put in a quick hack fix for the pong problem, but we're pretty certain that it's not a robust solution. We need to move to something more robust eventually.

See sections 5.4 and 5.5.x of the websocket RFC https://tools.ietf.org/html/rfc6455#page-36

Try to get declarative widgets working on top of the bleeding edge ipywidgets we're using here already.

/cc @lbustelo

Allow specification of the key and cert via nconf supported parameters.

In the dashboard notebook plugin, images in cells will resize to cell width:

In the dashboard server, images are always full size:

This is using all Markdown cells with <img> elements.

Gets confusing with lots of tabs. The filename would be fine. The first markdown header would be even better.

Add a notebook to this repo and instructions in the README about how to bring up an instance of the node app and have the notebook run as a dashboard. You'll probably want to use Docker because you'll need all of:

1. The notebook on disk
2. The node app
3. The kernel gateway
4. The kernel used in the notebook
5. Any other dependencies (matplotlib, node, npm, ...)

You could put all this in a single Docker container and launch two services for the demo (the node app + the kernel gateway). Or you could do a small docker-compose.yml with the kernel gateway running in one container (hopefully based on jupyter/docker-stacks#42 when it lands) and the node app in another.

The goal here is to show how the dashboard backend works independently of the bundler and the dashboard layout extension. Later, we can expand the README to link to those to paint the full picture.

TODO

• Create standalone demo notebook
• Create Docker container
• Replace jupyter-incubator/all-spark-kernels image with the minimal-kernel image.
• npm install jupyter-js-widgets instead of building it from source
• Use docker link command to link together the containers, instead of querying IP of kernel gateway container.

Strawman: some indicator in the nav bar at the top. Sort of like what notebook does when a kernel dies or a websocket disconnects or ... Don't hard code the indicator to kernel errors only though: it's very likely we're going to want to use that same indicator area to give other types of status too.

Should we hide hidden files/folders in the dashboard list? Simple fix to give a bit of polish.

It would be handy if the node app supported POST /notebooks to add / overwrite an existing notebook with the same with a new one for quick and dirty deployment.

https://github.com/jupyter-incubator/dashboards_nodejs_app/blob/master/routes/index.js

Let's not worry yet about persisting these or PUT vs POST or overwrite vs error. I want to get a simple demonstration of how a bundler can be written for Jupyter Notebook server to post a simple notebook to this API to get it to show up as a dashboard instantly.

The dashboard list page shows all contents of the data directory, which includes sub directories. However, selecting an entry in the list will attempt to load that as a notebook.

The GET /dashboards/* API does allow for sub directories, e.g. GET /dashboards/subdir/foo.ipynb.

Should we support sub directories in the list page or remove support from the GET API?

The ability to upload a notebook to the dashboards server does not allow for resources to go with it, such as widget files and data. Allow a zip file to be uploaded and its resources unpacked and served without clobbering the resources of other notebooks.

Structure: [updated 2016-02-26]

<notebook_name>/
    index.ipynb
    urth_components/
        ...
    stylesheet.css
    image.jpg

Client/Server Tasks:

• Support plain and bundled dashboards.
  • Plain & bundled should look same in UI.
  • Don't show ipynb extension.
• Support GET of bundled resources (everything but the notebook itself).
• Look for index.ipynb and always render for that dashboard for the dir, whether inside or outside of a "bundled" dashboard.
  • Look for index instead of __notebook__
• Load declwidgets from within bundled dashboard
• Create example installable through Makefile. Should test declwidgets, loading of local resources (CSS, images, etc).
• Support POST of bundled dashboards into data dir
• Create tests
  • Test loading index.ipynb (with/without extension). We should not get full contents back.
  • etc
• Update README to describe use of index.ipynb in a dir and local resources.

The current implementation assumes a standalone kernel gateway or notebook server. This is equivalent to tmpnb: false mode in thebe. It also needs to support the /api/spawn mode of operation to launch containers on tmpnb (or compatible) clusters.

Reason: tmpnb with its configurable-http-proxy has the only robust mechanism for reaping kernels around. A single kernel gateway is fine for testing, but it leaks kernels.

• call /api/spawn on tmpnb to request a kernel gateway (make sure each client talks to its own kernel gateway instance)
• support KERNEL_GATEWAY_URL and KERNEL_CLUSTER_URL properties (see comments)
• support passing tmpnb auth token to /api/spawn

Bottom left panel in https://github.com/jupyter-incubator/dashboards/blob/master/etc/notebooks/test/test_layout_basic.ipynb should have 3 linked widgets. Only one of them seems to render. The other two generate "Could not create view" errors in the JS console

Error: Could not create view↵    at new Error (native)↵    at Error.WrappedError (http://127.0.0.1:3000/components/jupyter-js-widgets.js:31415:26)↵    at promiseRejection (http://127.0.0.1:3000/components/jupyter-js-widgets.js:31493:34

Try https://github.com/jupyter-incubator/dashboards/blob/master/etc/notebooks/test/test_widget_overlap.ipynb in the notebook dashboard preview and then here in the dashboard server. The table styling from the notebook environment is missing here.

Related? When the table is wider than the window width, it does not overflow or scroll. In the notebook server, it overflows outside the allocated grid cell. The behavior should be consistent with the notebook.

vs

No console errors. Looks like something else broke gridstack. Confirmed that lodash is pinned. Even tried setting it to 3.10.1 exactly.

User lays out dashboard in the (mostly) fixed-width environment of the notebook server. We should match that experience when we display the notebook here. Depending on the browser width, the dashboard container width should snap to some maximum. (See the notebook CSS or the dashboards_bundlers CSS for what those breakpoints are.)

In #83, we decided to quick fix the styling problems by slapping the class on all output areas. This will probably have side-effects. We should revisit and find a way to apply it only when proper, when the output is actual HTML. This probably involves looking at the mimetypes in the display_data and execute_result messages in the dashboards.js#messageHandlers. At present, that function only receives a model. Adding a class dynamically would require a reference to the view as well. (Not sure such a reference is a good thing thinking about MVC design.)

Improve upon the immediate cleanup logic implemented in #25 to give the client time to try to reestablish its websocket connections.

Strawman: Set a timer in the nodejs app. If a client with the x-jupyter-session-id header returns before the timer fires, cancel the timer and continue forwarding messages to the associated kernel. If the timer goes off before a client reconnects, kill the kernel.

Of course, this also requires that the frontend JS attempt reconnections. It looks like jupyter-js-services already supports it, but we should check if it works in our use case.

Currently we have default bootstrap style login and dashboard list. If we do issue #87, we could follow up with login and list styling to match. I put these in a separate issue as I feel they are less important than the navbar since they reside in their own pages which are not the main focus of dashboards.

Try https://github.com/jupyter-incubator/dashboards/blob/master/etc/notebooks/test/test_html_javascript.ipynb in the dashboard server. Works fine in the notebook server and the old Thebe setup. Here, the %%javascript cell does not render as a script tag but rather as a javascript object string.

Might be a bug in jupyter-js-output-area or transformime. Might be a bug in how we consume messages.

Like the message did not reach the kernel in order. Or at all.

Support optional username and password configuration options via any nconf allowed method on server startup. If one is required, make sure the other is specified as well. If both are set, check whether or not all HTTP requests have a valid session cookie. If the user has authed (i.e., session cook is present and valid), service the HTTP request. If the user has not authed (i.e., the cookie does not exist, is expired, is invalid), show a basic login page prompting the user for the configured username and password. On successful login, set a temporary (1 day?) session cookie.

Use of client-side session cookies is fine here. (I think they're the express default still.)

Currently it's missing.

Most likely fix, call the jupyter-js-services shutdown method for the disconnecting websocket instance here: https://github.com/jupyter-incubator/dashboards_nodejs_app/blob/5ace686d7efd80eb1f70f3f8fc658c5a72504bf8/routes/api.js#L154 and in other errors cases too.

Thanks for your efforts on this repo ... I'm watching it with interest ... however:

I ran make run ... which appeared to succeed in building images and then launching containers ... I'm not sure if the bower install step didn't get run properly or something ... the terminal logs would suggest maybe it's missing express (see logs lower down)

The screen that is meant to list notebooks was blank ... and this is what shows up for the simple notebook.

-- Starting proxy container
npm info it worked if it ends with ok
npm info using [email protected]
npm info using [email protected]
npm info lifecycle [email protected]~prestart: [email protected]
npm info lifecycle [email protected]~start: [email protected]

> [email protected] start /home/node/app
> node ./bin/www

GET / 302 49.128 ms - 64
GET /notebooks 200 240.792 ms - 281
STACK: Error: Not Found
    at /home/node/app/app.js:106:15
    at Layer.handle [as handle_request] (/home/node/app/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/node/app/node_modules/express/lib/router/index.js:312:13)
    at /home/node/app/node_modules/express/lib/router/index.js:280:7
    at Function.process_params (/home/node/app/node_modules/express/lib/router/index.js:330:12)
    at next (/home/node/app/node_modules/express/lib/router/index.js:271:10)
    at /home/node/app/node_modules/express/lib/router/index.js:618:15
    at next (/home/node/app/node_modules/express/lib/router/index.js:256:14)
    at Function.handle (/home/node/app/node_modules/express/lib/router/index.js:176:3)
    at router (/home/node/app/node_modules/express/lib/router/index.js:46:12)
GET /sw-import.js?baseURI=http%3A%2F%2Flocalhost%3A3000%2Fbower_components%2Fplatinum-sw%2Fplatinum-sw-register.html&clientsClaim=true&defaultCacheStrategy=networkFirst&importscript=http%3A%2F%2Flocalhost%3A3000%2Fbower_components%2Fplatinum-sw%2Fbootstrap%2Fsw-toolbox-setup.js&precache=&skipWaiting=true&version=1.0 404 13.974 ms - 878
STACK: Error: Not Found

I tried to stop and restart the dashboard-proxy container, so that I might troubleshoot things manually inside the container ... but ctrl-c of the make run process appears to kill and remove that container.

I don't have that much experience with make, but each time a run that command, it appears to build a whole new image ... which obviously isn't as slow as the first time, as it uses caches etc ... but is there scope for getting this working using docker-compose, especially once the images are built?

When running make run-tmpnb I also hit a hurdle on it complaining that port 8000 is already being used ... Although I have other docker containers running ... none of them is touching port 8000 ... does this 1005/node on port 8000 look like anything related to this set-up?

Step 21 : CMD start
 ---> Using cache
 ---> 0e8b480a3420
Successfully built 0e8b480a3420
4eb7912c7e6fd15e096e18a8036bd476fd1c27c2dc831c084037d6286a6d3662
Error response from daemon: Cannot start container 4eb7912c7e6fd15e096e18a8036bd476fd1c27c2dc831c084037d6286a6d3662: failed to create endpoint tmpnb-proxy on network bridge: Error starting userland proxy: listen tcp 0.0.0.0:8000: bind: address already in use
make: *** [run-tmpnb-proxy] Error 1
mccoole@mccoole-ubuntu-W520:~/Development/Tools/jupyter/dashboards_nodejs_app$ netstat -tulpn | grep 8000
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      1005/node

• All params are nconf
• Expose IP interface as setting (default to 127.0.0.1)
• All params are documented in config.json and shown in make help
• Complete README sections
• make test within docker container
• Make cert targets real targets in makefile

Roughly here:

https://github.com/jupyter-incubator/dashboards_nodejs_app/blob/master/routes/index.js#L63

jupyter/dashboards_bundlers#12 will be the first client to use it.

(We're currently at 0.3.x)

Major change is that commOpened signal is gone, replaced with registerCommTarget().

Tasks:

• Replace use of commOpened with registerCommTarget().
• Fix display_data bug with jupyter-js-services (jupyter/services/pull/94)
• Update ipywidgets to use latest jupyter-js-services (jupyter-widgets/ipywidgets/issues/381)
• Update declarativewidgets to latest jupyter-js-services.
• Update declarativewidgets to latest ipywidgets (jupyter/declarativewidgets#157 (comment)).
• Don't build jquery-ui into ipywidgets, since we also use it and this leads to code duplication and bloat.

The impl in #1 will not be able to support ipywidgets or declarative widgets yet:

WRT ipywidgets support, there isn't support for jupyter-js-services yet. I'll probably end up moving to the jupyter-js-services comm API and shimming the current API (notebook compatibility), for ipywidgets 5.0 (next release) or 6.0 (depends on how much time I can allocate to 5.0).

From jupyter/dashboards#105 (comment)

Once the ipywidgets support for jupyter-js-services lands, the app here can start to take advantage of it.

Fix styling issues:

• Busy indicator appears at bottom of page, doesn't animate and doesn't disappear.
• Font sizes are different than dashboard in Notebook.
• Stream message outputs get appended as new elements in output area
  • When phosphor-widget is updated to 1.0.0-rc1 or later, re-commit 94d734d.
• Output area overlaps widget area, but only when running from Docker container (not from localhost).

We currently have a navbar with default bootstrap styling. We could style it to integrate better into the page and emphasize the page content.

• POST /notebooks -> move to POST /api/notebooks and keep as the programmatic way to add a new notebook
• GET / -> redirect and/or render /dashboards/index.ipynb if it exists or /dashboards if not.
• GET /dashboards -> UI list of all dashboards
• GET /dashboards/:name -> notebook rendered as a dashboard

The initial implementation should have the following properties

• fetch code from colocated notebook file on start
  • configurable path on disk
• use jupyter-js-services and jupyter-js-output-area, not thebe
• ability to filter websocket messages see issue #6
• supports one or more frontend browser client connections at a time and properly

Other issues will be opened about follow-on features.

We only use bower for Gridstack and its dependencies (jquery, jquery-ui, lodash). Should just simplify our build and install everything through npm.

Use https://github.com/jupyter-incubator/dashboards/blob/master/etc/notebooks/test/test_layout_basic.ipynb as a reference. In dashboard server, the h1 is smaller than the h2, for example. Headings not bolded the same way. And so on. Makes layout difficult to get right.

vs

1. tmpnb might be configured to require an auth token to request a KG server
2. KG server might be configured to require an auth token to request a kernel

Assume both of these keys can be set once at server start time via nconf, and just pass them along where appropriate. The keys should NOT be exposed to the frontend client.

• Pass auth token to kernel gateway
• Pass auth token to tmpnb
• Tests?

Current implementation prevents arbitrary code from the browser to reach the kernel. However, the browser can instruct the proxy to resend one of the code cells from a notebook multiple times. A bad actor might take advantage of this to mess with state on the backend if the notebook code is not written to account for multiple runs (e.g., incrementing a counter too many times).

Still, this is not a serious bug: the code already has to run once on the first load of the dashboard.

We have an ugly bit of code in api.js setupWSProxy() where we override socket.emit in order to add in the actual notebook code to be executed. It would be cleaner to use Node streams, if possible.

The http-proxy library does the proxying of websocket connections with the following code in ws-incoming.js:

proxySocket.pipe(socket).pipe(proxySocket);

I believe the socket variable above is what we pass in to the proxy lib in api.js:

proxy.ws(req, socket, head);

(not 100% sure about that, though).

If my assumptions above are correct, we should then be able to create our own Duplex Stream implementation that will change the incoming WS messages. In the end, we want the equivalent of:

proxySocket.pipe(wsRewriter).pipe(socket).pipe(proxySocket);

Since we don't care about the return WS msgs, we only need to insert our object (wsRewriter) on the incoming side.

Text cutoff in dashboard server, but fine in notebook https://github.com/jupyter-incubator/dashboards/blob/master/etc/notebooks/test/test_busy_indicator.ipynb.

For a deployed dynamic dashboard to be usable, people need to load an existing dashboard and immediately see its current state. For dashboards that are long-running (streaming) or that take time to run (Spark), it is not practical to re-run the code in a new kernel each time the page is loaded.

We should allow new connections to an already running kernel and properly build the dashboard with the required cell outputs and interactivity without having to re-run code.

Right now, it's a vanilla page with text that says "List of notebooks".

• Pretty it up to have the banner like the dashboard pages have when auth is enabled.
• Show a bootstrap styled list of dashboard links
• If there's a dashboard notebook called index.ipynb, redirect immediately to that instead of showing the list?

I'm now seeing a lot of these types of error in the server console:

STACK: Error: Not Found
    at /home/node/app/app.js:105:15
    at Layer.handle [as handle_request] (/home/node/app/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/node/app/node_modules/express/lib/router/index.js:312:13)
    at /home/node/app/node_modules/express/lib/router/index.js:280:7
    at Function.process_params (/home/node/app/node_modules/express/lib/router/index.js:330:12)
    at next (/home/node/app/node_modules/express/lib/router/index.js:271:10)
    at /home/node/app/node_modules/express/lib/router/index.js:618:15
    at next (/home/node/app/node_modules/express/lib/router/index.js:256:14)
    at Function.handle (/home/node/app/node_modules/express/lib/router/index.js:176:3)
    at router (/home/node/app/node_modules/express/lib/router/index.js:46:12)
GET /components/index.js.map 404 2.301 ms - 878

They don't really say much about what's going on. It just seems to be a generic error handler. @dalogsdon, I think we need to revisit how these error handlers are setup in the express app.

Go beyond the basic simple / web3. See what happens.

See title.

Kernel gateway supports a base_url parameter that allows it to run under non-root paths. This is necessary when it's deployed using something like tmpnb which runs multiple instances of the KG.

Make sure the dashboard server can:

• Proxy to a single kernel gateway that is running under a custom base_url
• [ ] Respects the URL returned by tmpnb's /api/spawn (see #11)

Port of the styling added to jupyter-incubator/dashboards and jupyter-incubator/dashboards_bundlers recently.