berkeley-dsep-infra / data8xhub Goto Github PK
View Code? Open in Web Editor NEWInfrastructure for creating & maintaining the data8x JupyterHub
License: BSD 3-Clause "New" or "Revised" License
Infrastructure for creating & maintaining the data8x JupyterHub
License: BSD 3-Clause "New" or "Revised" License
Currently, when the server dies, users see their code getting 'stuck', and not much else.
This is very bad UX!
We should instead have a proper notification that you can not connect to server. This should also differentiate between unable to connect vs an explicit 404 / 403, which indicates server has been culled. Should give indication to the user about what they should do!
Currently we do not have any CI/CD set up.
I think we should, but only for hub deployments. I think gdm changes should happen manually for now.
So they have instant feedback
We will have N NFS servers, each of which can serve a capacity Cn users. We need a high performance, non-racy and simple way of doing the following:
It also needs to perform at least some sort of authentication, split for reads and writes.
All this also needs to be fairly race free, and we should have a plan for backing up the data + reconstructing it from scratch if we lose it.
We want to have a bit more intelligent culling, so let's make the single-user notebook pods do the culling themselves with a notebook extension. This lets us kill things when kernels have been idle for a while, rather than network request based killing.
edX Students seem to be forgetting to save their work frequently. I suggest increasing the autosave interval to 5 minutes.
If the RAM gets close to what the limit is, start flashing!
Currently our secrets are uncommitted. Instead, we should be using git-crypt...
Collect metrics about things inside the cluster, with prometheus!
Should have a biggish SSD and a couple months retention.
We wanna do this!
Spin up a cluster with 50,000 pods separated out to say 10 namespaces.
And figure out how the cluster reacts to normal operations when under this much load!
If this proves to be too much, we can always split up our cluster into multiple smaller ones.
if we don't use ZFS RAID with multiple disks, we can simply rely on google's snapshots for backups. If we do use ZFS RAID, that one is on us.
If we can get away with non RAID'd performance, we should be ok to go without RAID.
We want a script that can take a notebook or a .py file, and push out a score. This score will be then sent to EdX
This is not all of them unfortunately, so we have to be careful to see that we engineer things from the beginning in a way that lets us change them over time.
Also try to finalize the config of these things before start of class.
Not sure if it ships with NFS mount support. If not, we need to switch to using ubuntu as the image for our k8s clusters.
Some of the content we use makes calls to external websites. These will be blocked per #11. We need to whitelist some domains through a HTTPS proxy.
We want our autoscaler to have some amount of 'resource slack' - it should expand the cluster not when it's 100% full but more like, 80% full.
Currently this isn't a supported feature, so we've to work around it in interesting ways.
Here's a proposal:
So sequence of actions here would be:
So the amount of headroom we have is equal to the 'minimum nodes' setting of the 'headroom' nodepool.
This also helps us not hit any github resource limits at scale.
We should probably be using https://github.com/kubernetes/git-sync + an nginx server.
The nfs-flex-volume provider is decent for what it does, but I don't trust the author so an alternative that's simpler would be nicer.
Kubernetes 1.10 brings rshared
mount propagation to beta, allowing us (possibly) to use it on k8s. However, that won't be ready in time, so we should do hacks that we trust to tide us over in the meantime. Ideally it'll be hostpath based...
When performance testing, I'm seeing the following errors when using DockerHub:
Warning Failed 49s kubelet, gke-hello3-cluster-alpha-default-d5650773-sxqq Failed to pull image "jupyterhub/k8s-network-tools:81c2613": pull QPS exceeded.
which is fair since it's a free service.
Does it? Does it work on IE? What do we do?
We want to capture NFS metrics from all our NFS hosts. We should do this via Prometheus, which seems to have a way of discovering Google Cloud services.
We shouldn't have tools for debugging on the main cluster. Instead we should have a 'support' cluster that contains things like:
This could be in a different zone too.
Currently the hub uses a local sqlite db.
Instead, use csql proxy and create a postgresql db per hub.
We need two kinds of backups:
(1) is first priority, (2) is not.
It looks like we'll end up with 30-40 hubs at max for this hub, so figuring out a load balancing strategy is important.
Requirements:
Currently, the homedirectory sharder is using a local sqlite database.
It should also be switched to a csql proxy + postgresql database
Make sure that users can't install new packages. We might even make the image readonly.
The user notebooks should have no access to the external internet at all.
They should have access only to the hub process, the proxy process and wherever we use to pull user content from.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.