twisted / txacme Goto Github PK
View Code? Open in Web Editor NEWTwisted client for the ACME (Automatic Certificate Management Environment) protocol
License: MIT License
Twisted client for the ACME (Automatic Certificate Management Environment) protocol
License: MIT License
There should be a twistd
command that runs an issuing service standalone. This would be most useful with http-01
and dns-01
challenge types, putting certificates into a store (eg. a directory) for other software to use. (It would be possible to make it work for tls-sni-01
as well, for that matter, but that's probably a niche use case)
In particular, route53 suffers from this.
Set up Towncrier, final documentation fixes, anything else?
This ticket is for tracking dns-01
support in general; #32 covers one specific backend for dns-01 challenges.
This is blocked on certbot/certbot#2061 which adds support for the dns-01
challenge to the acme
library, which is used by txacme
for all the protocol stuff. I'm not sure if any support code is needed in txacme beyond this and the IResponder
interface, but at the very least, there should be documentation about how to implement your own responder (eg. inserting records into the SQL database backing a PowerDNS deployment).
This is necessary, rather than going directly through the ICertificateStore
interface, because the issuing service may be in the process of reissuing the certificate, resulting in the new cert being inserted into the store after the removal, "resurrecting" it.
See #76 for the counterpart of this.
Because we may want to add things like #27 in future, I'm going to reserve all non-documented paths under the certs dir that the endpoint uses for future use.
Boulder doesn't support this yet, so it's not urgent, but we should do this.
This is one of the common DNS providers we should support. See also #45.
These are necessary for implementing dns-01
challenge responders, and probably useful in general.
This may or may not be something that txacme has to do anything about directly; it may be a documentation issue.
The overall error handling logic should mean that these errors are not fatal, but we ought to be able to retry immediately with the fresh nonce, which will make things a little more reliable.
To faster and support a secure adoption of the library it would be important to have txacme packaged for Debian/Ubuntu.
A good target could be probably to reach the Ubuntu xenial/Debian stretch that are just release so maybe there is still possibility for this.
In Client
there's not much ambiguity, but in the endpoint it's easy to see directory
and think it's what you want when you actually want cert_store=DirectoryStore(...)
.
These are deliberately compatible in the sense that you can point txacme
and txsni
at the same directory and have everything work (for example, web server with txacme
listening on port 443 to issue certs, and mail server with txsni
listening on port 25 sharing those certts).
The compatibility should be documented so that we ensure these remain compatible, and to clue in users about this synergy.
There were some changes in error handling, apparently:
[FAIL]
Traceback (most recent call last):
Failure: testtools.testresult.real._StringException: Traceback (most recent call last):
File "/home/travis/build/mithrandi/txacme/.tox/py27-twlatest/lib/python2.7/site-packages/txacme/test/test_client.py", line 1344, in test_check_invalid_error
failed_with(IsInstance(errors.ClientError)))
File "/home/travis/build/mithrandi/txacme/.tox/py27-twlatest/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: '(Error(detail=None, title=None, typ='about:blank'), TestResponse(code=403, content_type='application/problem+json', nonce=None, json=<function <lambda> at 0x7f06fefd4398>, links=None))' is not an instance of ClientError: after <operator.attrgetter object at 0x7f06fe655210> on <twisted.python.failure.Failure txacme.client.ServerError: (Error(detail=None, title=None, typ='about:blank'), TestResponse(code=403, content_type='application/problem+json', nonce=None, json=<function <lambda> at 0x7f06fefd4398>, links=None))>
txacme.test.test_client.JWSClientTests.test_check_invalid_error
This is a bit backwards incompatible for any error handling code, but it's unlikely this actually affects anyone.
This is going to be a tough one since acme
doesn't implement dns-01 yet (see certbot/certbot#2061) and there's no Route 53 implementation for Twisted anywhere.
Open questions:
Sometimes when I start up a twistd
using program that is using txacme (specifically the lets
, don't know if le
exhibits the same behavior) an error will get printed like:
2016-06-18T17:57:19+0000 [HTTP11ClientProtocol (TLSMemoryBIOProtocol),client] Unhandled Error
Traceback (most recent call last):
File "/usr/local/lib/pypy2.7/dist-packages/treq/client.py", line 62, in connectionLost
self.original.connectionLost(reason)
File "/usr/local/lib/pypy2.7/dist-packages/treq/content.py", line 38, in connectionLost
self.finished.callback(None)
File "/usr/local/lib/pypy2.7/dist-packages/twisted/internet/defer.py", line 393, in callback
self._startRunCallbacks(result)
File "/usr/local/lib/pypy2.7/dist-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "/usr/local/lib/pypy2.7/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/usr/local/lib/pypy2.7/dist-packages/eliot/twisted.py", line 82, in callbackWithContext
return self._action.run(callback, *args, **kwargs)
File "/usr/local/lib/pypy2.7/dist-packages/eliot/_action.py", line 381, in run
return f(*args, **kwargs)
File "/usr/local/lib/pypy2.7/dist-packages/txacme/client.py", line 710, in _got_json
messages.Error.from_json(jobj), response)
txacme.client.ServerError: (Error(typ=u'urn:acme:error:serverInternal', title=None, detail=u'Unable to update registration'), <treq.response._Response object at 0x0000000006f17ef8>)
This gets printed, and everything continues to run and it even appears to get certificates, however whenever you attempt to shut down the twistd
program using ctrl-c instead of shutting down a message like this grets printed:
^C2016-06-18T18:00:58+0000 [-] Received SIGINT, shutting down.
2016-06-18T18:00:58+0000 [-] Unhandled Error
Traceback (most recent call last):
File "/usr/local/lib/pypy2.7/dist-packages/twisted/application/app.py", line 390, in startReactor
self.config, oldstdout, oldstderr, self.profiler, reactor)
File "/usr/local/lib/pypy2.7/dist-packages/twisted/application/app.py", line 311, in runReactorWithLogging
reactor.run()
File "/usr/local/lib/pypy2.7/dist-packages/twisted/internet/base.py", line 1194, in run
self.mainLoop()
File "/usr/local/lib/pypy2.7/dist-packages/twisted/internet/base.py", line 1203, in mainLoop
self.runUntilCurrent()
--- <exception caught here> ---
File "/usr/local/lib/pypy2.7/dist-packages/twisted/internet/base.py", line 798, in runUntilCurrent
f(*a, **kw)
File "/usr/local/lib/pypy2.7/dist-packages/twisted/internet/base.py", line 581, in stop
"Can't stop reactor that isn't running.")
twisted.internet.error.ReactorNotRunning: Can't stop reactor that isn't running.
This happens once for each time you hit Ctrl-C, and the only way I've been able to shut down twistd
once this occurs, is to kill -9
the process.
If I start up twistd
without txacme I don't have a problem nor do I have a problem if I start it up and txacme manages to do everything without an error being generated.
This is going to involve deferToThread
and probably not be very "production-ready", but it has the advantage of not requiring writing a Twisted client library for some provider first (since apparently none exist currently), and providing reasonable support for a whole ton of providers at once. It also doubles as an example for implementing your own responder for, say, your internal PostgreSQL-backed PowerDNS or something.
See also #45.
https://txacme.readthedocs.io/en/latest/using.html says client_creator
could be, "for example", partial(Client.from_url, reactor=reactor, directory=LETSENCRYPT_STAGING_DIRECTORY, key=acme_key, alg=RS256)
. Except Client.from_url
doesn't take a directory
argument, so that's an error.
Currently, if you intended a service to be available for some name, but you forgot to put a cert for it into the store, the client will just get an inscrutable TLS failure, and nothing much useful will happen on the server end.
Is there something better we can do to clue the server admin in that they might be missing a cert?
Currently all ACME registrations are anonymous. Let's Encrypt doesn't have a problem with this, but some other future ACME CA might not allow this, and providing an email address with your registration enables you to receive certificate expiration warning emails from Let's Encrypt, which are useful if something goes wrong and you're not watching your logs closely to see the issuing failures from txacme
.
Importing any challenge type requires the libcloud
package to be installed. This should only be the case for the LibcloudDNSResponder
class, not for HTTP01Responder
and TLSSNI01Responder
.
>>> from txacme.challenges import HTTP01Responder
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/jamie/.virtualenvs/tmp-bcf6d230fbc44c61/lib/python2.7/site-packages/txacme/challenges/__init__.py", line 2, in <module>
from ._libcloud import LibcloudDNSResponder
File "/Users/jamie/.virtualenvs/tmp-bcf6d230fbc44c61/lib/python2.7/site-packages/txacme/challenges/_libcloud.py", line 7, in <module>
from libcloud.dns.providers import get_driver
ImportError: No module named libcloud.dns.providers
Authorizations may last for much longer than certificates, we should check to see if a new authorization is in fact needed before authorizing.
This hasn't been tested at all, there's probably some glitches here.
Currently, we use txsni, which allows for a DEFAULT.pem
, but txacme will try to issue a certificate for DEFAULT
when it tries to renew this, which will fail. We should have a better way for specifying which certificate to use for this.
Possibly related to #37 as both of these will require slightly more information than we are fitting into hostnames currently.
It's currently only possible to add/remove certificates from txacme by touching/deleting certificate files in the certificate directory (or adding/removing from some other store implementation).
It's technically possible to directly store empty certificates in an ICertificateStore
and then call the AcmeIssuingService._check_certs()
but that is messy and may be prone to race conditions with the timed invocation of the same method.
It should be possible to safely add and remove certificates from txacme.
A further iteration on this is perhaps the ability to modify certificates, e.g. to change (or as it is described in the certbot client, "expand") the SANs (#37).
$ mktmpenv
...
$ pip install txacme
...
Successfully installed Twisted-16.4.1 acme-0.9.3 attrs-16.2.0 cffi-1.8.3 cryptography-1.5.2 eliot-0.11.0 enum34-1.1.6 funcsigs-1.0.2 idna-2.1 ipaddress-1.0.17 mock-2.0.0 ndg-httpsclient-0.4.2 pbr-1.10.0 pem-16.1.0 pyOpenSSL-16.2.0 pyasn1-0.1.9 pyasn1-modules-0.0.8 pycparser-2.16 pyrfc3339-1.0 pyrsistent-0.11.13 pytz-2016.7 requests-2.11.1 service-identity-16.0.0 six-1.10.0 treq-15.1.0 txacme-0.9.0 txsni-0.1.6 zope.interface-4.3.2
$ twist --help
Traceback (most recent call last):
File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/bin/twist", line 11, in <module>
sys.exit(Twist.main())
File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/lib/python2.7/site-packages/twisted/application/twist/_twist.py", line 132, in main
options = cls.options(argv)
File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/lib/python2.7/site-packages/twisted/application/twist/_twist.py", line 38, in options
options.parseOptions(argv[1:])
File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/lib/python2.7/site-packages/twisted/application/twist/_options.py", line 157, in parseOptions
self.installReactor()
File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/lib/python2.7/site-packages/twisted/application/twist/_options.py", line 77, in installReactor
self["reactor"] = installReactor(name)
File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/lib/python2.7/site-packages/twisted/application/reactors.py", line 82, in installReactor
installer.install()
File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/lib/python2.7/site-packages/twisted/application/reactors.py", line 60, in install
namedAny(self.moduleName).install()
File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/lib/python2.7/site-packages/twisted/internet/selectreactor.py", line 198, in install
installReactor(reactor)
File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/lib/python2.7/site-packages/twisted/internet/main.py", line 32, in installReactor
raise error.ReactorAlreadyInstalledError("reactor already installed")
twisted.internet.error.ReactorAlreadyInstalledError: reactor already installed
This doesn't happen with twisted[tls]
, so I'm assuming it's txacme
's fault, but I haven't investigated much further than this.
Hi!
I think it might be useful to have a tutorial which covers "here are the steps you should take as someone who wants to set up a website with HTTPS on a new (sub)domain using txacme".
E.g., I am passingly familiar with letsencrypt, and with the letsencrypt binary, having used them a total of one time before discovering txacme. I'm also familiar with twisted.web, and specifically with endpoints. What I was missing though was a way to connect the dots. Sample outline, just based on the steps you just told me to perform:
(This is awesome by the way, looks like I'm quite close, once I remember my router password. Sigh.)
The IResponder
interface documentation says that for the start_responding()
and stop_responding()
methods, the challenge
parameter should be "The acme.challenges
challenge object".
The object passed to those methods in txacme.client.answer_challenges
is currently actually the acme.messages.ChallengeBody
challenge body.
This hasn't been a problem thus far because (@mithrandi on IRC)...
ChallengeBody has this:
def __getattr__(self, name):
return getattr(self.chall, name)
so mostly you can use the body interchangeably with the actual challenge, but not when calling .encode()
This does, however, cause a problem with the HTTP01Responder
as it calls .encode('token')
on the challenge
it is passed.
It'd be incredibly convenient if txacme would, when renewing a certificate whose name starts with www, request that the cert contain a subjectAlternativeName without the www
.
This is because of glyph/txsni#9, ideally this would just be fixed upstream but we may need to work around it ourselves.
Some error-handling changes in acme
mean that we are no longer at 100% coverage.
This would require enhancing the ICertificateStore
interface (IOpaqueCertificateStore
, maybe?); instead of txacme generating the private key and then signing the CSR with it, txacme would need to hand the CSR over to the certificate store for signing.
This is necessary for supporting HSMs, or HSM-like certificate stores (many software stores behave like an HSM in that access to the private key is restricted by policy, even though it obviously is still possible to extract the key).
Currently, a client object is passed in to AcmeIssuingService
; this means that the directory is only fetched once. However, the ACME service might have changes to the directory at any time; while such changes are unlikely to be frequent, the service should tolerate this happening during operation. We should probably start every client "session" by fetching the directory, thus a "client creator" should be passed in instead of a client object.
We should have these to go along with the other integration tests.
There should be documentation about implementing the ICertificateStore
interface. In particular, there is a nuance of this interface whereby you can handle keys in a "write-only" fashion; reading a certificate back only needs the certificate (to check the expiration date), not the private key, which is probably better security practice for networked implementations of this.
Twisted actually does have a sphinx inventory for the API docs, located at http://twistedmatrix.com/documents/current/api/objects.inv
This logic is quite useful when assembling an issuing service, so it should be split out as something standalone and public.
For example, for the dns-01
challenge, it is likely that different hostnames may live in different zones or even different providers, requiring different responders to handle. There should probably be a responder method that determines whether the responder can respond to a particular challenge, rather than the current hardcoded logic that only checks the challenge type.
Currently the code implicitly requires unicode ("text mode") paths, but there's nothing stopping you passing in a bytes mode FilePath
which will result in confusing errors.
There's a lot of ways to get an http-01 response into place, but writing it to a directory on the local filesystem is pretty commonplace.
We should really only call this in one or two places, and then pass the backend around. This would also pave the way towards allowing the user to pass in a backend of their own choosing.
Was getting errors when pip install Twisted[http2]
, does this work for you?
With the default client, a persistent HTTPConnectionPool
is created for connections to the ACME directory. This pool isn't tracked, however, and when stopService()
is called on the txacme service a pooled connection remains waiting on the reactor. This breaks my integration tests.
Something like txsni.maputils.Cache
is too simplistic, because it assumes the underlying store will never change, which is obviously false in the case of txacme.
The logic of making sure you clean up the challenge after you poll is left up to the caller, which sucks. There should probably just be a helper that combines answering and polling.
The only logs I'm getting are twisted.web.client factory messages. Txacme should at least log cert renewals and challenge responses.
No need for people to type these URLs out in the common (ie. Let's Encrypt) case.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.