ogrisel / docker-distributed Goto Github PK

Am I making a mistake with the setup?

docker run ogrisel/distributed start-dscheduler.sh
The `dscheduler` command has been deprecated.
Please use `dask-scheduler` in the future.

Traceback (most recent call last):
  File "/work/miniconda/bin/dscheduler", line 9, in <module>
    load_entry_point('distributed==1.12.0', 'console_scripts', 'dscheduler')()
  File "/work/miniconda/lib/python3.5/site-packages/distributed/cli/dscheduler.py", line 8, in go
    go()
  File "/work/miniconda/lib/python3.5/site-packages/distributed/cli/dask_scheduler.py", line 79, in go
    main()
  File "/work/miniconda/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/work/miniconda/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/work/miniconda/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/work/miniconda/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/work/miniconda/lib/python3.5/site-packages/distributed/cli/dask_scheduler.py", line 47, in main
    ip = socket.gethostbyname(host)
socket.gaierror: [Errno -2] Name or service not known

I just got my own Docker-distributed setup working in a vanilla form, then discovered you already had one. I'm still relatively new to Docker, and especially to Docker Compose and Swarm, so I have a few questions I wanted to run by you.

• Why a separate overlay network? To provide some context, networking is the single aspect of Docker I'm having the hardest time wrapping my head around, so almost every configuration that involves custom networking of any kind is foreign to me. Would love some intuition for what the custom overlay network does, and how that differs from the default network behavior created by Compose.
• Why is pshutil needed for the web front-end? That part seems to be working without any customization needed (with the obvious exception of exposing port 8787 in the Dockerfile).
• I'm probably still going to work just on my own setup, since I'm not really interested in using Kubernetes or hosting Jupyter notebooks. Instead, I'm looking almost at a sort of static Python job submission setup--spin up a Docker-distributed cluster, then SSH into the scheduler node and submit a Python script to run a job on the distributed dask cluster (who knows, may even create a tiny web front-end just for submitting Python scripts to obviate the need for SSH). Also, knocking out my own implementation will help me understand how it works!
• General comment: I am a big fan of how you used a single Docker image and then just ran separate bash scripts to execute different commands. Not sure why that didn't occur to me (I built three separate Docker images, which you can see in my repo linked above). I may refactor my own setup with that in mind.

Sorry if some of this got stream-of-consciousness, but I really enjoyed reading through what you've done so far.

@ogrisel
I noticed you added support for mounting a repo in your kubernetes set-up and I just wanted to understand better your workflow, as I'd like to be able to do something similar with a kubernetes implementation of jupyterhub.

When your notebooks and/or other files are updated while running on a container within kubernetes ... do you manually (maybe from a jupyter terminal) commit back to the repo? ... or is there some mechanism in kubernetes to automatically commit before a server session ends (preStop lifecycle hook)? Perhaps your use-case is just read-only?

Thanks.