Network protocol should support authentication about aenea HOT 17 OPEN

I could probably add the configuration option; it shouldn't be too difficult aside from how complex config is in Aenea, at least on the client side (hopefully at least the server side is clear).

Proposals: We should have both the client and the server authenticate to each other I think, rather than the one-way auth that's common, e.g., for a browser hitting a web page via https. I believe you can do this with SSL, but it's not something I've ever done. So the client and server each need their own public/private keypair, and the public key of the other. Those can be simple config options on both client and server, and we can default encryption to off if no key is specified. Note that some people have one client running on a windows box they use to remote control several servers, and so we may wish to address this. I'm fine with just having them use the same key on all the machines if you are. Alternatively, we could support multiple different keypairs, and possibly tie each one to a machine. I'm trying to remember if anyone uses multiple aenea clients to connect to one or more servers, and I don't think so. I would also be fine with just requiring one-to-one for authentication if you prefer; it's not a regression since we're adding an optional feature, and by general preference is to keep things simple.

It might also be nice to auto-generate pub/priv keypairs for users, thus requiring only that the user copy over the public keys to use encryption. Further automating key distribution makes me uneasy but I'm open to contrary opinions.

On a tangent, the reason config is so complex is because of the need to support a few different setups -- some people (myself included) are using a Windows VM for the client basically as a single-purpose embedded device: mine is not connected to the internet, and never does anything else. As such, I really want everything to be easy to configure from the Linux host, and don't care about strewing config files into C:/NatLink/...

Other people are using this on a real windows system or VM they want to use like a computer and keep neat and tidy.

These two use cases are somewhat at odds since the best way to address the first case is to keep all config in a network directory mounted on the host, and the second would move everything into a user directory somewhere (I don't really know much about Windows dev).

So my notion was to allow chaining of config files -- you just need to make a one-time change in the common denominator C:\NatLink... to point to where the real config file is, and then we follow it there. It's not a great solution for either case, but it works well enough for both, so I was happy.

I can certainly try to document this a bit better, you're not the first person to encounter this:-)

from aenea.

I had the same problem in one of my projects, unfortunately I solved it using python 3 libraries, so this will be of limited use here. Nonetheless, maybe an example will help: https://github.com/nihlaeth/all1input/blob/master/all1input/client.py

Ultimately I want to rewrite the above project to serve as an aenea client, but that's a distant dream for now.

from aenea.

I don't see a problem with moving the server to python 3, but I'm guessing I'd also need to move the client, which is deeply embedded in Natlink:-( Still, something to keep in mind.

from aenea.

I'd love to port natlink as a whole to python 3, and for the python part that wouldn't pose a big problem. It's the c part that worries me there…

As to using a 3 server and a 2 client, from the pip warning in python 2: 'A true SSLContext object is not available' whatever that entails, so I don't think the built-in libraries will be compatible with python 3's ssl module. pyopenssl could be a good alternative.

from aenea.

https://docs.python.org/2/library/ssl.html

Reading that, it shouldn't be too difficult. I'd be happy to make a pull request with the sockets wrapped in a TLS layer, and a script for creating certificates. But all this should be optional, and I'm really not at home with aenea's config system (if you're ever in the mood for documentation, that's one area that could use some). Do you want to work together on this?

from aenea.

I haven't forgotten!

With TLS, the server and client don't need each other's public keys ahead of time, the protocol takes care of the key exchange. Instead, the server and client both need a means to verify the signature on said keys. When you don't want to use expensive certificate authorities to have those keys signed, you can set up your own private certificate authority and sign your own keys. That sounds more involved than it is, it's a matter of a few simple commands that a script can take care of for you.

This way, both client and server need a certificate that is signed by your certificate authority to be able to connect to one another. When you add a new client or server to the network, you don't need to spread the new public key to all the existing clients and servers, you just need to sign the new certificate with the same certificate authority.

It would be great if you could add some config options for me. We'd need a switch to turn on SSL, and paths to the public and private key of the client/server, and the public part of the root certificate authority certificate.

from aenea.

Thanks for the explanation on how we can do this with a CA, that will work much better, especially for the multiple servers case. I believe I have implemented as you suggest in this branch: https://github.com/dictation-toolbox/aenea/tree/add_ssl. Let me know if I misunderstood and I can fix it.

On both client and server you should be able to use:
config.USE_SSL
config.SSL_PUBLIC_KEY
config.SSL_PRIVATE_KEY
config.SSL_CERTIFICATE_AUTHORITY_PUBLIC_KEY

I check that the paths to key material exist if USE_SSL=True and abort with a message if not, but I don't validate the files, just that we can read them.

Aside: That configuration code is simply atrocious. The server is tolerable, hard coding values in config.py is not ideal but at least simple enough to understand. The config code on the client side seems to be one of a few holdovers from this project's origin as a dirty hack. This took me much longer than I originally thought and there are some compromises in it. Hopefully enough to unblock you at least.

I really do need to turn this into a proper Python project (or pair of projects) at some point. Separate everything out and test what I can, proper configuration and construction, come up with a clean way to share code between client and server, and assimilate everything into using logging properly come to mind. This just sort of grew organically out of a hack I never planned to share with anyone into something that other people actually use. I've been incrementally rewriting (it used to be much worse, I promise!), but clearly there is still substantial work to be done.

from aenea.

Awesome! I added a certificate creation script to the project, I wrote a little bit in the readme about how to use it. Now all that's left is actually wrapping the socket.

In the readme I left a spot for some configuration instructions. Would you fill that?

If you want a second voice working on refactoring and packaging, I'm available. Just say the word. :)

from aenea.

Could you shed some light on the client socket situation? I'm looking at line 81 because it looks like a promising place to wrap the socket, but as far as I can see this socket is never used or stored.

from aenea.

Note to self: make a version of this class that inherits from SafeTransport. I think that should enable SSL, though I still don't get what that socket from my previous comment is doing there.

Edit: original SafeTransport

from aenea.

Line 81 is just testing the connection before creating the transport as a bit of a hack. I THOUGHT I remembered this predating the ImpatientTransport thing, yet looking at the git history that predates it? I'd need to look into it more to figure out if it's still needed, HOWEVER I'm pretty sure that once the connection succeeds the socket is discarded and not actually used.

Which raises the question of whether or not we need to wrap it. Since its purpose is to test the connection, ideally we'd do the handshake to exercise the crypto too. Only worth the trouble if it's easy.

All this messing with timeout stuff was trial and error to deal with irritating pauses in dictation caused by hitting the server at inopportune times in callbacks from Natlink. We got it to a state where it mostly worked and more or less left it alone.

from aenea.

Ok actually I think I remember -- it's defense in depth, since with dictation even timeouts of 0.05s or so can cause problems (I can elaborate if necessary).

Basically, before connecting we attempt to just open a socket with a super short timeout. If that fails, no point in continuing to try. Dragon is frozen whenever we're blocked in client code, and we may need to do this whenever you talk.

Then, once we can open a socket, we open the ImpatientTransport with a slightly more generous timeout (as too short a timeout of course causes its own issues, especially for our colleagues using this over a real network rather than just a VM bridge).

So it's not just an artifact of the past, we should keep it.

As to wrapping, I have changed my mind (that was fast) and am leaning against it -- I want to keep that pre-check as fast as possible, since it occurs whenever you speak (with a slight cooldown). If the connection succeeds, we should proceed right away to using the Transport. If it fails, there's no point. The only thing wrapping it would do is detect connection working but SSL not, and it's not worth the trouble to handle that, since that's a configuration error (or attack:P) rather than a simple timeout or whatnot.

from aenea.

Thanks for the explanation. No wrapping, but if I remember correctly, socket will throw an exception when attempting to connect to an SSL enabled socket unwrapped, so I'll have to catch that.

from aenea.

On a different note, the need for hacks like this is exactly why I'm itching for asyncio. I've been looking at the natlink source, and I'm starting to believe that porting to python 3.6 is doable.

from aenea.

@nihlaeth

First off - thanks for taking the time to dig into some of this!

On a different note, the need for hacks like this is exactly why I'm itching for asyncio. I've been looking at the natlink source, and I'm starting to believe that porting to python 3.6 is doable.

Short of a natlink re-write I don't think that blocking io itself is the issue here. The aenea wants to know its execution context before performing any action. Regardless of whether or not the underlying io is async we're going to wait for the get_context function to return prior to performing the next action in the chain. Right now we cheat a little bit by caching these contexts but that comes with its own issues.

Another thing to look into here is to make sure that we're using some sort of keep-alive for our RPCs (I don't remember if we currently do this or not). Especially for those that are using aenea over anything other than a host/vm network. Creating new sockets is slow, creating new tls sockets is even slower!

Speaking of speed... I saw that you contributed to the xdotool linux server implementation. If you're feeling adventurous then give the libxdo implementation a go. Input may be quite a bit faster for you.

python server_x11.py --input=libxdo

Note: You'll need the libxdo-dev package if you're running an Ubuntu based distro.

@calmofthestorm

Aside: That configuration code is simply atrocious. The server is tolerable, hard coding values in config.py is not ideal but at least simple enough to understand. The config code on the client side seems to be one of a few holdovers from this project's origin as a dirty hack. This took me much longer than I originally thought and there are some compromises in it. Hopefully enough to unblock you at least.

Testing non desktop environment specific server functionality should be relatively straight forward after the re-write that enabled pluggable implementations. It'd be cool to see some of that work finally pay off :-)

from aenea.

@mzizzi you're welcome :-)

Short of a natlink re-write I don't think that blocking io itself is the issue here. The aenea wants to know its execution context before performing any action. Regardless of whether or not the underlying io is async we're going to wait for the get_context function to return prior to performing the next action in the chain. Right now we cheat a little bit by caching these contexts but that comes with its own issues.

I'm planning on doing just that, completely rewriting the Python part of natlink. I want to turn the whole context thing upside down, and have the server communicate context changes to Dragon as they happen, instead of on request. I don't know yet in what sort of timeframe I could get this up and running though.

Another thing to look into here is to make sure that we're using some sort of keep-alive for our RPCs (I don't remember if we currently do this or not). Especially for those that are using aenea over anything other than a host/vm network. Creating new sockets is slow, creating new tls sockets is even slower!

I have no experience with jsonrpc, so I don't know how they handle their sockets. In the past I've left sockets open indefinitely, and only reconnect after short sleep if the connection is interrupted or dropped. Sockets can stay open for weeks at a time like that without problems.

from aenea.

I know very little about Natlink, so I don't have specific thoughts on the feasibility of inverting context (so to speak), but if you can get it working that would be a huge improvement. get_context has a ton of hackery and is a major drag on responsiveness.

from aenea.