upip doesn't appear to validate HTTPS connections about micropython-lib HOT 19 OPEN

As @Lukasa notes

No, he doesn't. At the very best, he hints, but mostly he bikesheds around the idea "as there exist orthogonal problem of key distribution (which applies to any key-based security model), a scheme which protects content can't be called better that the one which protects merely a transport channel", which is a strange argument. Nick, you continue to bikeshed around, even though besides the main topic of this ticket (upip doesn't verify certs), there're only couple of very focused issues were brought up:

• Security is end-to-end thing. Sticking something at the center doesn't offer complete security.
• And if there's problem on the last mile, like a user doesn't want all that security baggage, then there won't be security.
• Then unconditionally forcing security-in-the-middle isn't very wise, especially if everyone else allows to turn off real-security on the last mile (obviously, at the user risk) to enable testing, debugging, and "innovative uses" (tm).

Anyway, the issue of undue forcing SSL where a user doesn't want it was successfully resolved, on the last mile mention above. The remaining issue is using security aspect SSL when user really wants that. Let the rest of the discussion be dedicated to that.

from micropython-lib.

micropython-upip 0.7 with the warning pushed to PyPI and micropython source tree, to be builtin in the upcoming 1.8.1 release.

from micropython-lib.

The full CACert bundle is on the order of ~300 KB: https://github.com/certifi/python-certifi/blob/master/certifi/cacert.pem

I realise that's quite large in the context of MicroPython, so one possible way to go might be to disallow HTTPS connections (at least in upip.download, potentially in ussl.wrap_socket itself) if:

1. The generic certificate bundle is not available; and
2. No more specific certificate information is provided

That way, HTTPS can be made to "just work" on larger devices with room for the generic certificate bundle (and a way of securely updating that bundle), without presenting a misleading impression as to the default level of security on smaller devices.

from micropython-lib.

micropython/micropython#1862 covers the concerns with ussl skipping certificate verification by default, even after PEP 476 switched CPython to verifying by default (the solution used in CPython unfortunately won't work for MicroPython, as it relies on system certificate stores being available)

from micropython-lib.

Thanks @ncoghlan for the report, and sorry for no reaction. @pfalcon and I are now gearing up for a few months of deep coding on the ESP8266 port, and one of the last stretch goals for that project was proper upip support on the ESP itself. So we will get to this issue by then :)

from micropython-lib.

Sorry for the delay with the response - it actually took some time to monitor our progress to come up with an answer which would include realistic promises (or dismiss them based on objective, not subjective state of affairs).

Reading https://www.modio.se/presentations/Intro_to_TLS_pycon_2016.pdf reminded me of an issue and hopefully now a response would be (at least somewhat) grounded:

the way ussl works correctly, the default behaviour is to skip certificate verification
This is a problem not only for ussl, but also for upip specifically

This is true, and by design. By the design of initial proof-of-concept implementation of upip, and ussl module was added with upip as the main client first of all.

Let me start with saying that I shared my concerns of unduly enforced SSL when accessing PyPI with distutil-sig: https://mail.python.org/pipermail/distutils-sig/2014-November/025237.html . That exchange went pretty short, as by that time I already figured that upstream distutils-sig isn't interested in helping MicroPython (or any other constrained Python implementation) to access/adapt/reuse existing "big Python" tooling and infrastructure, and any effort in that direction would be on MicroPython side.

My initial idea was to develop, in Python, a module called insecureSSL, the minimal possible SSL implementation which would not just ignore certificate stuff, but used zeros, empty, and other static values as various nonces, keys, etc. To prove the point, the point being that you can't secure someone who doesn't want to be secure. Because MicroPython at its initial stage, current stage, and probably for some more time doesn't need any SSL on pypi.python.org at all.

As can be seen, I decided after all to use this motion in a direction more beneficial for MicroPython, and provide integration and binding for an actual SSL library. However, it is being used per the original intent: specifically, upip accesses HTTPS on pypi.python.org as if it was plain HTTP.

What's interesting, "you can't secure someone who doesn't want to be secure" goes both ways. Because in its current form, PyPI is "very secure" potential trojan galore. It should be very obvious how it works: someone uploads "dgango" package with the proverbial "format c: /y", done. So, it's very ironical that PyPI takes a patronizing stance of requiring even the little ones to access it with SSL, while providing far from complete overall security.

To turn this rant into a case study, let me show that the above is not my wishful thinking, or worse, hallucinations, and that it's indeed possible to do it differently, and the industry indeed does it.

Xively

https://personal.xively.com/dev/docs/api/security/ , bottom:

While it is possible to communicate with Xively using HTTP, this method is not secure and it is not recommended. It remains a part of the service as an element of legacy support. It is recomended to use ‘HTTPS’ in all API requests: https://api.xively.com.

That's it. Even no red font (I would use it). As a result, Xively service is really helpful not just for productions usage, but also for testing/debugging and new device bring-up.

Debian (and other Linux distributions)

Unlike PyPI, Debian provides no-nonsense security with signed packages. And yet the package manager has options (--force-yes, etc.) to bypass it if a user so desires.

Now back to the issue of verifying certificates. Should it be done, in general? Yes. Is it an issue that it's not done? Yes. So, this ticket should stay open and wait for its stakeholders to gather and come up with a solution to the problem.

from micropython-lib.

Just to be clear, signed packages is not helpful in a model like PyPI's. As an example of their lack of utility, let me point out to you that the h2 module signs its releases. What information does validating that signature tell you about the security of the package?

from micropython-lib.

Just to be clear, signed packages is not helpful in a model like PyPI's.

Yes. And my point is that overcompensating that with forcing SSL isn't the brightest solution either. Even providers of signed packages, whose security model is much stronger, allow to disable it as a user discretion.

from micropython-lib.

My point is that signed packages is not a stronger security model unless you have a solution to key distribution. Most package distribution mechanisms do not.

from micropython-lib.

As @Lukasa notes, the benefit of forcing HTTPS isn't in the use of SSL per se, it's in being able to rely on the Mozilla cert bundle to shift your threat model from "anyone on the link between you and the server" to "nation state level adversary that can compromise the CA trust network".

PEP 458 and 480 shift that model to being anchored in a PyPI published key that can be bootstrapped via HTTPS on a system other than the consuming device, but the CA system is still the ultimate arbiter of trust (and likewise for Linux distros, now that those tend to be bootstrapped via ISO, VM and container image downloads, rather than physical media shipped by the vendor).

Aside from all that though, embedded devices absolutely, 100% should NOT be downloading software directly from the public internet - compose an atomic bundle on a trusted build service, secure the link between that service and the device however you want, and only install those pre-built components. Doing systems integration work directly on the target device has proven to be a bad idea, which is why people are also moving away from it for conventional Linux systems (with things like ChromiumOS, CoreOS, Project Atomic, and Ubuntu Snappy Core).

While I haven't looked into the specific security details of their model, https://resin.io/ is a system that at least gets this right at the conceptual level.

For a less monolithic approach, the Python ecosystem offers https://github.com/cloudmatrix/esky/ which could likely be adapted to manage bundled virtual environments and/or zip archives in addition to full frozen applications.

from micropython-lib.

Paul, if you want a security model other than HTTPS to secure content passed to embedded devices, then all you need to do is write a caching proxy server that implements your alternate protocol while using HTTPS to talk to PyPI on the backend. There is nobody forcing you to use SSL on low powered endpoint devices unless you insist on accessing PyPI directly without going through a caching proxy.

That approach has the further benefit of providing the actual security model that Linux distros use to get around the key distribution problem: the same entity will be responsible for the security of both the endpoint devices (embedded systems running MicroPython for you, installed OS instances for distro publishers), and for the software publishing system (caching proxy for you, the package build system for distro publishers).

But if you're not willing to invest the time needed to do that yourself (especially given projects like devpi that already handle the caching proxy part), please don't try to push the responsibility for that decision to opt out of secure software distribution onto any other project.

from micropython-lib.

However, we have veered significantly from the original point: it is NOT OK to silently fail to validate HTTPS certificates by default in a project claiming conformance with either Python 3 or pip without the user explicitly opting in to that lack of verification. This is a "truth in advertising" matter - the standard library and pip offer certain default guarantees, so anyone emulating them needs to honour those guarantees to avoid misleading their users.

However, it is 100% fine to add explicit opt-out capabilities to turn off certificate validation (even when the reference implementations don't offer that), since CPython and pip are both targeted at environments where full SSL/TLS support (including cert validation) can be assumed, while the same isn't the case for MicroPython and upip.

Given that MicroPython can't rely on system certificate stores, I'd even consider it reasonable for MicroPython to merely issue a warning if no validation certs are configured, with two options for getting rid of it (either adding validation certs, or explicitly opting out of cert validation)

from micropython-lib.

unless you insist on accessing PyPI directly without going through a caching proxy.

Yes, that's what I want, and I consider that part of (implicit) social contract: MicroPython project tries to apply and reuse as much as possible upstream Python groundwork instead of carving up its own cozy kingdom and reinventing various kinds of wheels. It should be pretty clear why it benefits community. And why I consider it beneficial personally, as a developer, is that before working on MicroPython, I worked on other scripting languages, which lacked a lot of infrastructure, and I had full freedom to do it like I want. But turned out that designing and planning various interfaces and infrastructure is a lot of work, and coming to work on MicroPython, I made explicit choice to avoid reinventing something, unless proved to be necessary.

please don't try to push the responsibility for that decision to opt out of secure software distribution onto any other project.

Well, that's what has happened - opt out happened on upip side.

from micropython-lib.

in a project claiming conformance with either Python 3 or pip

Well, we don't claim anything like that - MicroPython implements (arbitrary) subset of Python3, and upip is just a workalike to pip.

the standard library and pip offer certain default guarantees

You mean, after many years of having various issues, they are finally were made to work more or less well.

Given that MicroPython can't rely on system certificate stores, I'd even consider it reasonable for MicroPython to merely issue a warning if no validation certs are configured, with two options for getting rid of it (either adding validation certs, or explicitly opting out of cert validation)

I can queue up adding such warning to upip, which is an interactive application, so such output won't harm. Module "ussl" was documented as not validating certificates, per micropython/micropython#1862 (comment) , so there's no false advertising.

from micropython-lib.

A warning from upip would mitigate most of my immediate concerns, as one of the key points we need to keep in mind when negotiating these user experience details is the difference in threat models and operating assumptions for CPython and MicroPython.

For mainstream Python and pip, the main threat vector of current concern is open Wi-Fi networks (whether at conferences, coffee shops, airports, hotels, or elsewhere). For those, validated HTTPS connections are a strong countermeasure. PEPs 458 and 480 will add to that basic privacy and integrity assurance, but they won't replace it.

MicroPython is a more specialised scenario, where the main trap we want to avoid encouraging people to fall into is the one that afflicts many industrial control systems: heavy reliance on perimeter defence to create a trusted network, with few or no security measures on the endpoint devices themselves. As more and more systems communicate over the public internet rather than air-gapped (or at least firewalled) networks, the notion of a "trusted network" is becoming less and less viable, even in industrial control settings.

from micropython-lib.

MicroPython is a more specialised scenario, where the main trap we want to avoid encouraging people to fall into is the one that afflicts many industrial control systems:

Formulated like that it's certainly ok. But for avoidance of doubt I'd like to say that MicroPython may not yet be suitable for "industrial control systems". And the obvious reason would be that industrial control systems industry didn't contribute to MicroPython to make it such. But with a message formulated as "we can do better than "many industrial control systems", by at least not pretending we're secure where we aren't", I can certainly agree.

from micropython-lib.

Warning added in 6cd414e.

from micropython-lib.

Thanks! As time goes by the investment needed to help build more secure package update mechanisms for MicroPython will hopefully be forthcoming, but a visible warning is a reasonable compromise until that happens.

from micropython-lib.

As I mentioned above, this ticket represents a real issue, and would rather stay open (to serve as a gathering point for parties interested in resolving it). If you aren't interested to receive further updated, please unsubscribe from it (button on the right).

from micropython-lib.