Wildcard certificate is not loaded from storage about certmagic HOT 18 CLOSED

Note: it seems like the in-memory cache is checked with many level of wildcard (*.example.com then *.*.com). As far as I know this is not necessary as TLS certificates can only have a single left-most level of wildcard.

Same thing for the storage, only one possible wildcard needs to be checked as far as I know.

from certmagic.

Hm, okay, yeah, so I think this only makes sense for On-Demand.

As a fallback, the storage is then checked. However, only the exact match for foo.example.com is attempted.

I suppose we could make this an optional behavior, a boolean field in the OnDemandConfig struct; if enabled, try loading wildcard permutation of the domain name. We probably shouldn't assume that a wildcard should always be used if that's what is available -- usually, on-demand TLS and wildcards are not used together, so I'm curious about your need for this.

Of course, this raises the question of what to do if a cert does not yet exist for that domain. Currently, CertMagic will try to get a cert for the exact domain, not for the wildcard variant of it (i.e. a handshake for foo.example.com will try to get a cert for foo.example.com not *.example.com), so unless we did that too, there'd be an asymmetry. Acceptable? If not, we need to figure out the implications of managing only wildcard certs instead of specific SANs.

In general I feel like this is a bad idea, as the DNS challenge (which is required by Let's Encrypt for wildcards) is the slowest of all the challenge types, and not optimal for use in the foreground as on-demand TLS is.

Note: it seems like the in-memory cache is checked with many level of wildcard (*.example.com then ..com). As far as I know this is not necessary as TLS certificates can only have a single left-most level of wildcard.

RFC 5280 leaves it up to the application; 6125 restricts it to left-most label. As a server that manages and chooses certs from a pool, we should be more liberal in what we accept from the sysadmin, but clients should be more conservative when verifying certificates.

from certmagic.

Thank for the quick reply!

I suppose we could make this an optional behavior, a boolean field in the OnDemandConfig struct; if enabled, try loading wildcard permutation of the domain name. We probably shouldn't assume that a wildcard should always be used if that's what is available -- usually, on-demand TLS and wildcards are not used together, so I'm curious about your need for this.

Make sense to me. There is indeed not point at adding a perf pernalty if you already know you don't have wildcard certificates.

I don't need on-demand TLS here, I just want to load the existing certificates from the storage. But it seems that CertMagic doesn't do that unless on-demand TLS is enabled (https://github.com/caddyserver/certmagic/blob/master/handshake.go#L234).

Of course, this raises the question of what to do if a cert does not yet exist for that domain. Currently, CertMagic will try to get a cert for the exact domain, not for the wildcard variant of it (i.e. a handshake for foo.example.com will try to get a cert for foo.example.com not *.example.com), so unless we did that too, there'd be an asymmetry. Acceptable? If not, we need to figure out the implications of managing only wildcard certs instead of specific SANs.

For the record, I have both exact domain and wildcard in this storage, for different usage.

In general I feel like this is a bad idea, as the DNS challenge (which is required by Let's Encrypt for wildcards) is the slowest of all the challenge types, and not optimal for use in the foreground as on-demand TLS is.

Agreed. This is one of the reason why those wildcard certs are generated by another process. The other reason is that the system that knows what cert are needed is not serving traffic himself. Those certs are generated ahead of time, ready to be used by an actual node. At least that's the plan.

RFC 5280 leaves it up to the application; 6125 restricts it to left-most label. As a server that manages and chooses certs from a pool, we should be more liberal in what we accept from the sysadmin, but clients should be more conservative when verifying certificates.

If I'm not mistaken, some browsers (firefox) strictly adhere with 6125 and will simply refuse such a certificate. That should leave very few possible use cases for such a feature IMHO. Just my 2 cents.

from certmagic.

Just a FYI, I will also need a way to renew those certificates, either by those nodes serving traffic (I suspect the cache does that in the background) or by this centralized system. I haven't figured out what's the best way to do that yet.

from certmagic.

I don't need on-demand TLS here, I just want to load the existing certificates from the storage.

In that case, just pass *.example.com as one of the domains into Manage*().

For the record, I have both exact domain and wildcard in this storage, for different usage.

So, pass foo.example.com and *.example.com both :)

If I'm not mistaken, some browsers (firefox) strictly adhere with 6125 and will simply refuse such a certificate. That should leave very few possible use cases for such a feature IMHO. Just my 2 cents.

Sure, that's by design. There are many internal clients that benefit from alternate validation logic. Not everything is in the public trust.

Just a FYI, I will also need a way to renew those certificates, either by those nodes serving traffic (I suspect the cache does that in the background) or by this centralized system. I haven't figured out what's the best way to do that yet.

Just let CertMagic do it. When it's loaded into the cache (as a managed certificate, basically any method of loading that doesn't involve calling CacheUnmanagedCertificate()) it will be maintained.

What is this "centralized system" you are working on?

from certmagic.

In that case, just pass *.example.com as one of the domains into Manage*().

I can't. I don't know the full list of domains beforehand, and more domains can (will) be added while the node is running.

What is this "centralized system" you are working on?

In short it's a user management system. A bunch of domains exist per user, each of those needs a certificate. When a user is added, new certs need to be created and added in the storage, ready to be picked by the nodes serving traffic.

from certmagic.

If these are your users' domains, how are you able to get wildcard certificates with the DNS challenge? Do you actually control the DNS for them?

from certmagic.

I'm using the DNS challenge indeed. It's a single domain I fully control. I need certificates per user because of the single level wildcard limitation.

Would it be a problem for CertMagic to have thousands of certificate in the storage ? (I'm aware of the LetsEncrypt rate limiting).

from certmagic.

I'm using the DNS challenge indeed. It's a single domain I fully control. I need certificates per user because of the single level wildcard limitation.

Can you spell out a few examples then? If it's one domain, how many domains could there possibly be that you don't know about?

Would it be a problem for CertMagic to have thousands of certificate in the storage?

No. It gracefully handles tens of thousands of certificates, as long as you have enough RAM + CPU (hardware-accelerated crypto, etc).

from certmagic.

In short I want to have domains in the form of <contentID>.<userID>.example.com. It would be trivial if multi-level of wildcard were possible but it's not.

Enter CertMagic and LetsEncrypt. I can instead generate a wildcard cert per user and leave the contentID open.

I know this might looks like a crazy/stupid idea but that's the only possible way I have due to other constraint. It's this or simply not do this project.

from certmagic.

In short I want to have domains in the form of ..example.com. It would be trivial if multi-level of wildcard were possible but it's not.

Thanks for clarifying, that is very different from your original post. :(

Well, as a temporary workaround, one easy solution is to just update CertMagic's config with the new <userID>.example.com domains when you know what they are.

from certmagic.

I have 3 ideas for the on-demand config:

• A simple bool (all on or off for all domains)
• A list of right-most-side of domains for which to get left-most-label wildcards for (i.e. a list like example.com would match foo.bar.example.com and thus would result in a cert for *.bar.example.com)
• A special return value from the decision func (or the "ask" endpoint if you are familiar with Caddy) that specifies the actual name to get a cert for.

I think I'm leaning toward the second one.

from certmagic.

Well, I simplified the problem. In reality I need a dozens certificates per user, half of them being wildcard. Those include public and internal domains, for different environments (dev, staging, prod). It's fairly impractical for me to drive CertMagic from the "node serving traffic" side, as that would involve a complex system just to update the config on those nodes each time a change happen on the user list.

It seems to me that the feature I'd like to see in CertMagic (being able to pre-generate certs in a system, use them from another) is different than on-demand TLS. Trying to fit that into the on-demand config doesn't work so much.

On the other hand, the storage provide a great abstraction and enable new possible ways to use CertMagic, like my contraption. It would just need some additional config to define what sort of certificates can be attempted to load from the storage to avoid the performance penalty in normal cases. Pushing the concept further, it would be even possible to add a more complex query to the Storage interface to load the different combination in a single call (but I don't need that).

from certmagic.

It seems to me that the feature I'd like to see in CertMagic (being able to pre-generate certs in a system, use them from another) is different than on-demand TLS. Trying to fit that into the on-demand config doesn't work so much.

So maybe you just need a Caddy config like this:

{
	"apps": {
		"tls": {
			"certificates": {
				"automate": [
					"*.sub.example.com",
					"example.com"
				]
			}
		}
	}
}

See, you don't even have to run an HTTP server.

Then just grab the certs from whatever storage you configure (default = file system).

from certmagic.

We already have a custom HTTP proxy for plenty of other reasons 😬. Adding another layer of proxying would make things way too complicated and possibly just break stuff.

I understand if you don't want to make this change. If you don't I can just have an internal fork with the few lines to load the wildcard certificates from the storage if missing.

However, as a software designer, it looks to me that adding a few options to configure the loading from the storage would really enable some nice use cases. If you want, I can try to come up with a
more precise proposal.

from certmagic.

We already have a custom HTTP proxy for plenty of other reasons 😬. Adding another layer of proxying would make things way too complicated and possibly just break stuff.

This is not a proxy. This is only a process that manages certificates for your domains.

I understand if you don't want to make this change.

Which change? I'm not convinced that a change needs to be made at this point.

If you want, I can try to come up with a more precise proposal.

Yes, precision would be helpful, since right now I'm not sure what the problem is anymore. It seems to have changed.

from certmagic.

So you are not suggesting to have Caddy handle all the TLS stuff ? I'm a bit confused there.

Ok, I'll come up with a proposal.

from certmagic.

So you are not suggesting to have Caddy handle all the TLS stuff ? I'm a bit confused there.

What are you confused about? The config above obtains and manages certificates for those domains. It doesn't have to be your HTTP proxy too if you already have one.

But it makes sense to have fewer moving parts, so... anyway it's up to you.

Would be happy to read through a more detailed proposal.

from certmagic.