This code base to deploy Jupyter Notebooks using JupyterHub is based on the open source reference implementation from https://github.com/jupyterhub/jupyterhub-deploy-docker.
It also uses Docker (https://www.docker.com/) containers to manage the three pieces of software needed to run this set up:
- JupyterHub - takes care of authentication and notebook spawning
- Jupyter Notebooks - notebook environment Python, R and Javascript kernels, plus BeakerX kernels (includes Scala and Java)
- PostgreSQL - database backend to store notebook user data
Git clone https://github.com/PHI-Toolkit/jupyterhub-deploy-docker-localhost. Change to the jupyterhub-deploy-docker-localhost
folder and run the steps below.
On Ubuntu Linux 18.04 (Bionic), you might have to execute the following (Linux terminal commands):
- Install git -
sudo apt install git
- Install Docker, Docker Compose, make -
sudo install-docker-bionic.sh
(see more below, "Install Docker, Docker Compose and make on your remote Virtual Machine (VM)")
The next steps entail three rounds of modification levels (1=no modification (default settings), 2=LetsEncrypt certificates, 3=GitHub OAuth authentication).
This first round build enables you to run as localhost
on your laptop with a self-signed certificate (https://localhost
). If you set it up on a remote VM, i.e., Digital Ocean, Lightsail, Linode, Contabo, you can use the VM IP address instead of localhost
.
To run the first build with no modifications:
- Run
time buildhub.sh
(Thetime
beforebuildhub.sh
gives you total script run time. This script takes approximately an hour, aside from seeing commands being executed on screen, you may want to grab some cofee and do other things.) - When the script finishes, run the
starthub.sh
script - Go to https://localhost on your Chrome browser to view the JupyterHub log in page. The default login name is "jovyan" and password is set in the
.env
file. If on a remote virtual machine, replace "localhost" with the machine's IP address. If you have mapped the VM IP address to a domain name, you can replace the IP address with the domain name. You should see a similar GitHub sign in page as below.
To enable GitHub autentication:
- Edit the
.env
file and change theJUPYTERHUB_AUTHENTICATOR
the default valuedummy_authenticator
togithub_authenticator
- Go to GitHub and obtain OAuth credentials (see below for details, "Obtain your GitHub Account Credentials"). Register for a GitHub account if needed, and obtain your GitHub username.
- Plug in the values for
GITHUB_CLIENT_ID
,GITHUB_CLIENT_SECRET
, andOAUTH_CALLBACK_URL
in the corresponding variables in the.env
file. - IMPORTANT: Edit the
userlist
file and add your GitHub user ID to the line belowjovyan
and addadmin
after your user ID. This enables you to use your GitHub user ID to authenticate via GitHub. If you missed this, you may get the Browser 403 error (see below). - Run the
stophub.sh
script if the Jupyter Notebook server is still running. You can check if the hub and single user containers are running withdocker ps
. - Run the
time buildhub.sh
script. - Run the
starthub.sh
script - Go to https://example.com. You will be redirected to the GitHub site to allow
hermantolentino
to autenticate you and to fill in your GitHub user ID and password. You will be redirected back to the Jupyter Notebook server after that. You should see a similar GitHub sign in page as below.
Modify .env
file LetsEncrypt section:
- Get a domain name (e.g., example.com) from a domain registrar. Make the necessary changes so the DNS of the domain registrar points to the remote VM hosting the Jupyter Notebook server. IMPORTANT: Propagation of the domain name throughout the Internet may take a few hours, so if you intend to use a domain name (needed for LetsEncrypt set up) it is highly recommended to get it set up early, perhaps before beginning Round 1.
- Fill up
JH_FQDN
,JH_EMAIL
andCERT_SERVER
variables in the.env
file. LeaveCERT_SERVER
blank. - In the
.env
file, change the value ofJUPYTERHUB_SSL=use_ssl_ss
(default) toJUPYTERHUB_SSL=use_ssl_le
to use LetsEncrypt to generate SSL certificates. - Run the
stophub.sh
script if the Jupyter Hub server is still running. - Run
sudo time buildhub.sh
script. You needsudo
to run theletsencrypt-certs.sh
with admin privileges. - Go to https://example.com/ (just an example of domain name) and authenticate as in Round 1. You should see a valid LE https certificate in the URL as below.
Git clone https://github.com/PHI-Toolkit/jupyterhub-deploy-docker-localhost. Change to the jupyterhub-deploy-docker-localhost
folder and run the steps below.
- Run the
install-docker-bionic.sh
to install Docker and Docker Compose on a VM running Ubuntu Bionic. - If the VM is not running Bionic Beaver, run the
upgrade-distro.sh
script. Be sure to replace the source and destination OS in thesed
statement. - Make sure to exit the VM and log back in for changes to take effect. Run
docker images
to check if you have access to docker without sudo. If runningdocker images
generates an error, runsudo usermod -aG docker [replace-with-your-username]
.
References
- Install Docker: https://docs.docker.com/install/
- Install Docker Compose: https://docs.docker.com/compose/install/
This build of JupyterHub has three options for Authentication. Go to about Line 18 of the .env
file and set the environment variable JUPYTERHUB_AUTHENTICATOR
to the selected option.
- tmp_authenticator
- dummy_authenticator (default)
- github_authenticator (thru OAuth, requires obtaining GitHub credentials, see below)
Possible scenarios for GitHub authentication (if you choose 'github_authenticator'):
-
Default GitHub settings in .env - no need to change or update any .env settings
-
Set up your own GitHub account and OAuth:
2.1 Set up GitHub account
2.2 Set up OAuth application (see below "Obtain GitHub Credentials")
If you will be using GitHub Oauth to authenticate users to JupyterHub, you need to sign up for a GitHub Account:
- Go to https://www.github.com and create an account if you do not have one yet.
- Remember your GitHub user name. You will use this for #3 below.
- Open the file
userlist
with your text editor and add your GitHub user name below "jovyan admin" as below:
<github user name> admin
- Log in to GitHub
- Go to Developer Settings (https://github.com/settings/developers) - create new Oauth App
- Record the following information:
- GitHub Client ID
- GitHub Client Secret
- GitHub Callback URL: This should be of the form https://"mydomain.com"/hub/oauth_callback if with a domain name (remember to replace "mydomain.com" with your domain name, as obtained from the step above, "Using a Domain Name".)
- Copy-paste each of these to right
.env
section (about Line 23):
GITHUB_CLIENT_ID=<github client id>
GITHUB_CLIENT_SECRET=<github client secret>
OAUTH_CALLBACK_URL=https://mydomain.com/hub/oauth_callback
If using localhost, replace "mydomain.com" in OAUTH_CALLBACK with "localhost" (i.e., "https://localhost/hub/oauth_callback").
Run sudo time buildhub.sh
.
Upon runnng starthub.sh
, you will likely see an error message saying you need to run jupyterhub upgrade-db
. You can do this by:
-
Running the
upgrade-jupyterhub-db.sh
script against a running container -
Commenting lines 71-72 of the
docker-compose.yml
, and uncommenting lines 73-74 (runsjupyterhub upgrade-db
), then runningdocker-compose build
. After running the database upgrade, runstophub.sh
again, uncomment lines 71-72, comment out lines 73-74, and then runningdocker-compose build
again. Then runstarthub.sh
. (based on https://jupyterhub.readthedocs.io/en/stable/reference/upgrading.html)
-
If using custom Dockerfile, use the following values:
a. DOCKER_NOTEBOOK_IMAGE=jupyter/minimal-notebook:459e68c2f8b7
b. DOCKERFILE=Dockerfile.custom -
If using Docker Jupyter Stacks (https://github.com/jupyter/docker-stacks), use the following values:
a. DOCKER_NOTEBOOK_IMAGE can be any of the following:jupyter/scipy-notebook:459e68c2f8b7
,jupyter/r-notebook:459e68c2f8b7
, orjupyter/datascience-notebook:459e68c2f8b7
b. DOCKERFILE=Dockerfile.stacks
Windows 10 users should do the following:
- To run the bash scripts in a bash shell on Windows 10, install the Linux Subsystem on Windows 10 here: https://docs.microsoft.com/en-us/windows/wsl/install-win10
- Use "Edge Channel" Docker version for Windows 10: https://download.docker.com/win/edge/Docker%20for%20Windows%20Installer.exe
- Check "Expose daemon on tcp://localhost:2375 without TLS as follows:
- While jupyterhub is running, type the following commands:
$ docker exec -it jupyterhub /bin/bash
- This brings you to the jupyterhub bash prompt. Type the following command to regenerate a new cookie secret:
# openssl rand -hex 32 > "/data/jupyterhub_cookie_secret"
- Add your GitHub username to the
userlist
file as described above.
- If you see this error in the logs it means the JUPYTERHUB_SERVICE_HOST_IP is misconfigured.
- There may be slight differences in how Chrome or Firefox behaves compared to installations on Linux or Mac (YMMV).
If you get the error:
An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way. ConnectionError(ReadTimeoutError("HTTPSConnectionPool(host='conda.anaconda.org', port=443): Read timed out.",),)
...and the buildhub.sh
script is building the user container image (running Dockerfile.custom), just run make notebook_image
to resume rebuilding the Jupyter Notebook user container image.
This error could be due to JUPYTERHUB_SERVICE_HOST_IP changing value after restarting Docker server (after server reboot). To address this error, run the script get_service_host_ip.sh
at the command line, which will provide you with the new JUPYTERHUB_SERVICE_HOST_IP value. Replace the old value in the .env
file with this new IP address. Run restarthub.sh
.