pipalacademy / pipalhub Goto Github PK

In a markdown cell, with a code block opened by three ticks, a space before the first newline triggers an error from nbconvert. If we remove the space. it works fine!

{three ticks}{space}
a  3
b  4
{three ticks}

Right now every participant need to login to the notebook with credentials. It would be nice to generate auto login link and use that to login the user to jupyter hub without login.

We should have some browser js provided with the dashboard service that can check to see if the page content needs updating.
This can be done with polling GET /events and comparing that with the timestamp of last fetched event.
For now, the entrypoint to this script should be an checkForUpdate(filters: Record<string, string>). Filters would usually be on either filename (for summary) or path (for individual notebooks).
This should return a promise that resolves when an update is available.

Usage should be like:

checkForUpdate({filename: "module1-day1.ipynb"})
    .then((saveEvent) => {
        notify(`Please refresh. ${saveEvent.path} has been updated.`);
    })

There might be an optional interval parameter for how frequently the endpoint is polled.

Q: Should checkForUpdate be renamed to waitForUpdate? Gives a nice flow with wait().then()

The user should receive some notification about comments put by trainer.
either notification is needed or at least count of unseen comments should be visible
at Problem Set level (just like you shoe Submitted!)

Export notebooks on save instead of polling and building them every few seconds.

This would replace the build scripts in the pipalhub setup. That means we also need to do ipytail and generate summary on every save.

It should have two endpoints:

• A webhook that should be hit when a file is saved
  • Data should have:
    • which user performed the save
    • the filesystem path of the notebook
  • The service will then figure out how to use this information
• An endpoint that will return which notebook was the last to be saved and when.
  • This can be polled to figure out if there has been a new change.

Webhook:

> POST /events/save
> {
>    "user": "alice",
>    "path": "/home/alice/module1-day1.ipynb",
> }
>
...
< 200 OK

Polling endpoint:

> GET /events/save/module1-day1  # /events/save/{path/to/file/after/home} ?
>
...
< 200 OK
< {
<     "timestamp": "2022-11-14T06:19:03Z",
<     "user": "alice"
< }

Jupyterhub services reference: https://jupyterhub.readthedocs.io/en/stable/reference/services.html

This will make it easier to start the jupyterhub and programatically setup users without having to create a users.txt file.

We can use the /events endpoint implemented in #6 to start creating summary pages when a notebook is saved.
The summary page is likely to change heavily, but it'll have the same approach - perform some action when a notebook-save event is received.

TODO: create an on_save(event) function in dashboard service that will take an event of type "save-notebook" and perform some action.
For now, this action should be to use the existing code from build container to write the summary pages.

This should remove the need for a build container.

It would be good to have an admin interface to manage pipalhub instances.

The admin should be able to:

• login with google auth
• see home page with list of active instances and a buttton to add a new instance
• The

Login

Only authenticated users should be allowed to access this system. Login with google auth should be supported and anyone with @pipal.in email should be allowed to login.

Home page

The home page lists all the active instances and have a link to add a new instance.

For each instance, it should show the name, hostname, size, image and owner.

New Instance

The app provides an ability to create a new instance of pipalhub.

It would ask for the following:

• name (hostname will be $name.pipal.in)
• size (mapping of size names to digitalocean instance sizes will be specified in the config file)
• python version (mapping of python version to docker image tag is specified in the config file)
• runtime -- name of the runtime to use (runtime captures the custom packages to be installed, specified in the repo)
• label - a text tabel to identify the name

This would automatically setup the node.

Instance page

This would show the following information about the instance:

• name
• hostname
• size
• python-version
• runtime
• label
• list of users
• creation timestamp
• created by / owner

This will also have the following actions:

• restart the node
• add new user / delete user / rename user
• update (run install script again)
• delete instance

This should also have an link to show the system logs.

This should also have a log of activity, showing all the changes that are made to this instance (including add user, update instance, restart node etc.)

Right now, it takes too many steps to setup a new instance of pipalhub. We need to make it into a couple of steps.

Adding users and installing custom packages is out of scope for this issue.

Proposed Approach

Assuming that you have created a new server and logged into it as root.

Assume that the node is Ubuntu 22.04.

$ curl https://github.com/..../setup-node.sh -o /tmp/setup-node.sh
$ bash /tmp/setup-node.sh lab.pipal.in

This would setup everything and the server would be ready to use.

Improvement 1

We could supply those two lines as initialization script (userdata) when creating a new node so that we don't have to run that manually.

Improvement 2

Create a create-node.py that takes DIGITALOCEAN_TOKEN from env and creates a new node of given size and hostname.

This is run from any developers node.

$ export DIGITALOCEAN_TOKEN=abcd1234
$ python create-node.py --hostname lab.pipal.in --instance-size small|medium|large

This should add all the ssh-keys in the account to the node.

This should also create a DNS record for the hostname.

When the users are added to the jupyterjub, the shell defaults to sh instead of bash.

For example, the following are the entries in /etc/passwd file in the jupyterhub container.

kaustubh1:x:1000:1000::/home/kaustubh1:/bin/sh
kaustubh2:x:1001:1001::/home/kaustubh2:/bin/sh
anand:x:1002:1002::/home/anand:/bin/sh

This leads to starting sh when a terminal is opened from jupyterlab.

I think we need to fix this by explicitly specifying the shell when adding users.