Comments (23)
Maybe remote kernel can be used/integrated to support this feature?
from hydrogen.
I would still like to support this.
@rgbkrk, I think you had some concerns about zmq on random ports making it through people's networks?
@TKCen it looks like rk
is helping write kernelspecs for remote systems. If they work with Jupyter, I would expect them to work with Hydrogen. The only caveat is that if you have a local kernel for a particular language, Hydrogen may use that instead (it uses the first matching kernel it finds).
Would someone be willing to give rk
that a try and let us know if it works?
from hydrogen.
Just opened #39 to address switching between multiple kernels for a language.
from hydrogen.
rk
appears to be over SSH which works for a lot of folks, myself included. The "template" kernel would probably work just fine as is. Might as well try it out. I wouldn't want to manage dependencies this way, but it's at least simple for most users. It doesn't seem like it would require any special requirements on Hydrogen at all.
As for my concerns, these are my observations from teaching classes, running a few tmpnb deployments, JupyterHub deployments, and my own security perspectives. Feel free to take it all with a grain of salt.
Remotes over ZeroMQ
It's all about the kernels communication layer. Not every ZeroMQ installation includes encryption by default (libsodium is an optional dependency for ØMQ). ØMQ4 does have curve, which is wire-compatible with ØMQ3, but won't necessarily have libsodium. One can build it that way for themselves, certainly. This tends to mean either using websockets (through a notebook server) or tunneling through SSH.
Remotes over websockets
The most reliable way I've seen for getting websockets through is making sure they're served over HTTPS on the standard port (443). That boils down to corporate (MITM) proxies totally mucking up the sticky connection with websockets, interfering with the flow, and generally being nasty. As for port, that's about stringent controls (no high ports getting through, etc.) Use HTTPS and you're fine!
Upstream!
That being said, I think the remote kernel issue should be addressed with upstream Jupyter projects to work for the notebook server, etc. As long as we're adhering to the same protocol, it should help multiple projects at once.
I see a near future where folks can consume kernels remotely from wherever they have them deployed, whether that means O'Reilly Media providing them for static content (e.g. https://beta.oreilly.com/learning/an-illustrated-introduction-to-the-t-sne-algorithm, through the use of tmpnb + thebe), a JupyterHub deployment (complete with auth), or some other spawning mechanism. There's not a current authentication construct/config around providing support for remote kernels (from a project level) other than IPython parallel and full notebooks with the JupyterHub spawners, but I'd like to see it soon.
from hydrogen.
Interesting — if something like rk
works, it's probably an OK stopgap measure. We can revisit this when Jupyter adds more features for remotes that will let this be a good experience (and not leave the kernel server sitting wide open).
from hydrogen.
I tried rk, Jupyter connects to the kernel momentarily, then loops between "Dead Kernel" and "Connected" every twenty seconds or so.
from hydrogen.
Forgive my ignorance, is the security issue with the remote kernel only relevant in open networks? Lots of use cases involve only talking to closed networks through a vpn or some such.
from hydrogen.
Forgive my ignorance, is the security issue with the remote kernel only relevant in open networks? Lots of use cases involve only talking to closed networks through a vpn or some such.
For someone advanced enough to do that, they can already use remote kernels just fine. It's the launcher that has to perform it. The kernel.json spec file could specify whatever IP you bind to:
{
"stdin_port": 57985,
"ip": "127.0.0.1",
"control_port": 61088,
"hb_port": 55956,
"signature_scheme": "hmac-sha256",
"key": "1fab9810-6e36-40df-a400-ceb196fbf9a7",
"shell_port": 54211,
"transport": "tcp",
"iopub_port": 56874
}
The default in the notebook is to do localhost so that we're not exposing typical users to remote execution of their laptop while, e.g. at a cafe.
from hydrogen.
Can someone please give us some example on how to set this up? I am a total noob when it comes to ipython kernnel
-ing, and a bit noob with ssh
as well (I am connecting remotely through VPN). I'm trying to figure out where this kernelspec should go... I see a folder Hydrogen/kernel-configs
but something tells me that those are created each time a kernel is run through atom
. How do you do this thing more concretely? A page in the wiki would be nice! Thanks
edit:
OK, so I successfully followed the recipe here to run a kernel on a remote server and connect to it using ipython console
through ssh
, but I have no idea how to instruct Hydrogen to use the server... if someone could clarify this would be great!
from hydrogen.
There's no documentation because this is definitely not supported behavior right now. Since Hydrogen is designed to start the kernel itself rather than connecting to an existing kernel, it's going to be weird if it works at all. Hydrogen currently connects via local ZeroMQ ports. Furthermore, Hydrogen does not use any security at all, since it's designed for local access, meaning that the server and probably also your local machine would have a huge security hole.
Honestly, I think you're going to be better off using ipython
itself for now. I definitely want to support this usage at some point, but there are a lot of changes that need to be made between now and then (both to Hydrogen and to Jupyter). Building remote systems is hard!
I'm sorry I don't have a better answer for you right now, but keep an eye out for changes to come.
from hydrogen.
I will post shortly... I have just made this to work (without changing anything in hydrogen
). So thank you guys! now I have interactivity with my remote kernel which is awesome!
from hydrogen.
Right on!
On Wed, Jul 1, 2015 at 12:30 PM Răzvan Rădulescu [email protected]
wrote:
I will post shortly... I have just made this work (without changing
anything in hydrogen)—
Reply to this email directly or view it on GitHub
#38 (comment)
.
from hydrogen.
My major issue with not using message signing, even though local, is that anything else that can reach those ZMQ sockets can get remote execution. They have to know the port, but it's not like they can't port scan.
If someone got it to work through the flash binding (zmqsocket.js) you would have a XSS attack with full remote execution from random sites on the internet. That's not good. There's a reason the key and signature scheme is provided.
from hydrogen.
But it sounds to me that there's a super easy solution to this... You just build the ability to add remote machines in hydrogen
, where you specify the remote path to the kernel-1234.json
file that the ipython
creates on the remote kernel. Read that file and include the key
information in the configuration, then do the HMAC serialization... Obviously, this would mean that you need to start the ipython
kernel yourself on the remote machine, but I don't see this as a problem.
from hydrogen.
@rgbkrk Ugh, yeah. Let's get signing working.
@razvanc87 Yep — signing and remote kernels are two different steps.
I agree that getting it just to the point of running in Hydrogen isn't a ton of work. The harder steps about making that use case work as smoothly as possible for people.
from hydrogen.
If you're interested in working on this, pull requests are very much appreciated :)
from hydrogen.
Yep, I definitely am, but I have a lot of ground to cover... I have no experience with this coffeescript
stuff, and even less with zmq
!... but anyway, until then, I have put this gist together explaining how to set the remote communication up... maybe you guys can take a look and let me know if it makes sense, if I need to change anything or complete/modify. Thank you!
oh right... I forgot to explain what the bash script does... oh well :))... details
from hydrogen.
Awesome work on this — I'll direct people this way in the future!
from hydrogen.
Any news on this?
from hydrogen.
@oxinabox I haven't used them myself, but I'm aware of, at least, three projects to provide remote kernels. See:
- https://bitbucket.org/tdaff/remote_ikernel
- https://github.com/korniichuk/rk
- https://github.com/danielballan/remotekernel
@rgbkrk @lgeiger Currently, the only way to start up Hydrogen is to make a run request. Functionality like switching kernels is not available before that. I think we could provide two more commands:
- Start kernel (this would make the current switch command obsolete)
- Connect to running kernel (this would prompt for a connection file)
from hydrogen.
We've got a couple remote kernel setups working as part of enchannel, though they're not ready for utility consumption.
from hydrogen.
old link form @razvanc-r was not working anymore, link to his gist is: https://gist.github.com/razvanc-r/a365b921ba4bf575b6a5
from hydrogen.
Closed via #295
from hydrogen.
Related Issues (20)
- hydrogen fails to install on Atom HOT 2
- fialedto activate
- npm WARN deprecated [email protected] HOT 2
- Failed to activate the Hydrogen package HOT 18
- [email protected] failed
- Interactive input in output area
- VSCode: port or alternative HOT 12
- Uncaught Error: Cannot find module 'lodash/debounce'Require stack:- $ATOM_HOME/packages\Hydroge... HOT 3
- Uncaught error: no native build was found for platforms HOT 4
- Please port to Vs Code or any other IDE HOT 1
- Uncaught Error: No native build was found for platform=darwin arch=x64 runtime=electron abi=85 uv... HOT 2
- TypeError when using hydrogen with pulsar HOT 4
- Hi, If the notebook in "not trusted" mode. Just click on it and make it trusted then save the file. It will work fine.
- SCRIPT WONT EXECUTE. HOT 1
- can't rebuild zeromq for pulsar
- Newly installed python modules cannot be found by hydrogen in pulsar
- [18:30] r p npm WARN deprecated [email protected]: this library is no longer supported [18:31] r p npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic [18:31] r p npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic
- deleted
- Is this going to be maintained? HOT 5
- Publish Hydrogen to Pulsar Package Repository HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hydrogen.