Comments (19)
Ksdmg
commented on August 10, 2024
I think I have the same Issue, here are my logs:
boulder-boulder-1 | 2022-10-27T12:03:54.277717+00:00 263e8a95e41c unknown health-checker[2384]: 3 health-checker 2ZDr_wk [AUDIT] timed out waiting for va1.boulder:9098 health check
boulder-boulder-1 | 2022-10-27T12:04:21.050869+00:00 263e8a95e41c unknown health-checker[493]: 3 health-checker z_v_qQ8 [AUDIT] timed out waiting for publisher1.boulder:9091 health check
boulder-boulder-1 | 2022-10-27T12:04:48.191218+00:00 263e8a95e41c unknown health-checker[470]: 3 health-checker z_v_qQ8 [AUDIT] timed out waiting for publisher1.boulder:9091 health check
boulder-boulder-1 | 2022-10-27T12:05:14.354641+00:00 263e8a95e41c unknown health-checker[469]: 3 health-checker kYynsQE [AUDIT] timed out waiting for nonce2.boulder:9101 health check
boulder-boulder-1 | 2022-10-27T12:05:41.681563+00:00 263e8a95e41c unknown health-checker[503]: 3 health-checker 0tf3rQo [AUDIT] timed out waiting for nonce1.boulder:9101 health check
boulder-boulder-1 | 2022-10-27T12:06:08.619053+00:00 263e8a95e41c unknown health-checker[477]: 3 health-checker r_m5yww [AUDIT] timed out waiting for va1.boulder:9097 health check
boulder-boulder-1 | 2022-10-27T12:06:35.779158+00:00 263e8a95e41c unknown health-checker[488]: 3 health-checker jKCvtQQ [AUDIT] timed out waiting for publisher2.boulder:9091 health check
boulder-boulder-1 | 2022-10-27T12:07:02.284289+00:00 263e8a95e41c unknown health-checker[497]: 3 health-checker tNz0_wo [AUDIT] timed out waiting for sa2.boulder:9095 health check
boulder-boulder-1 | 2022-10-27T12:07:29.8611+00:00 263e8a95e41c unknown health-checker[501]: 3 health-checker jKCvtQQ [AUDIT] timed out waiting for publisher2.boulder:9091 health check
boulder-boulder-1 | 2022-10-27T12:07:58.66596+00:00 263e8a95e41c unknown health-checker[411]: 3 health-checker jKCvtQQ [AUDIT] timed out waiting for publisher2.boulder:9091 health check
boulder-boulder-1 | 2022-10-27T12:08:24.220767+00:00 263e8a95e41c unknown health-checker[405]: 3 health-checker tNz0_wo [AUDIT] timed out waiting for sa2.boulder:9095 health check
boulder-boulder-1 | 2022-10-27T12:08:49.659519+00:00 263e8a95e41c unknown health-checker[416]: 3 health-checker tNz0_wo [AUDIT] timed out waiting for sa2.boulder:9095 health check
boulder-boulder-1 | 2022-10-27T12:09:14.688871+00:00 263e8a95e41c unknown health-checker[399]: 3 health-checker kYynsQE [AUDIT] timed out waiting for nonce2.boulder:9101 health check
boulder-boulder-1 | 2022-10-27T12:09:39.775116+00:00 263e8a95e41c unknown health-checker[399]: 3 health-checker z_v_qQ8 [AUDIT] timed out waiting for publisher1.boulder:9091 health check
boulder-boulder-1 | 2022-10-27T12:10:05.141438+00:00 263e8a95e41c unknown health-checker[405]: 3 health-checker tNz0_wo [AUDIT] timed out waiting for sa2.boulder:9095 health check
boulder-boulder-1 | 2022-10-27T12:10:31.679168+00:00 263e8a95e41c unknown health-checker[410]: 3 health-checker tNz0_wo [AUDIT] timed out waiting for sa2.boulder:9095 health check
boulder-boulder-1 | 2022-10-27T12:10:57.00838+00:00 263e8a95e41c unknown health-checker[410]: 3 health-checker z_v_qQ8 [AUDIT] timed out waiting for publisher1.boulder:9091 health check
boulder-boulder-1 | 2022-10-27T12:11:22.923165+00:00 263e8a95e41c unknown health-checker[410]: 3 health-checker tNz0_wo [AUDIT] timed out waiting for sa2.boulder:9095 health check
boulder-boulder-1 | 2022-10-27T12:11:47.953284+00:00 263e8a95e41c unknown health-checker[394]: 3 health-checker 0tf3rQo [AUDIT] timed out waiting for nonce1.boulder:9101 health check
boulder-boulder-1 | 2022-10-27T12:12:13.177431+00:00 263e8a95e41c unknown health-checker[399]: 3 health-checker tNz0_wo [AUDIT] timed out waiting for sa2.boulder:9095 health check
boulder-boulder-1 | 2022-10-27T12:12:38.43599+00:00 263e8a95e41c unknown health-checker[411]: 3 health-checker jKCvtQQ [AUDIT] timed out waiting for publisher2.boulder:9091 health check
boulder-boulder-1 | 2022-10-27T12:13:05.137111+00:00 263e8a95e41c unknown health-checker[389]: 3 health-checker tNz0_wo [AUDIT] timed out waiting for sa2.boulder:9095 health check
boulder-boulder-1 | 2022-10-27T12:13:33.25763+00:00 263e8a95e41c unknown health-checker[399]: 3 health-checker tNz0_wo [AUDIT] timed out waiting for sa2.boulder:9095 health check
boulder-boulder-1 | 2022-10-27T12:14:01.49593+00:00 263e8a95e41c unknown health-checker[410]: 3 health-checker r_m5yww [AUDIT] timed out waiting for va1.boulder:9097 health check
boulder-boulder-1 | 2022-10-27T12:14:29.772359+00:00 263e8a95e41c unknown health-checker[412]: 3 health-checker 2ZDr_wk [AUDIT] timed out waiting for va1.boulder:9098 health check
boulder-boulder-1 | 2022-10-27T12:14:57.782317+00:00 263e8a95e41c unknown health-checker[399]: 3 health-checker 0tf3rQo [AUDIT] timed out waiting for nonce1.boulder:9101 health check
boulder-boulder-1 | 2022-10-27T12:15:27.031331+00:00 263e8a95e41c unknown health-checker[394]: 3 health-checker z_v_qQ8 [AUDIT] timed out waiting for publisher1.boulder:9091 health check
boulder-boulder-1 | 2022-10-27T12:15:55.930488+00:00 263e8a95e41c unknown health-checker[388]: 3 health-checker jKCvtQQ [AUDIT] timed out waiting for publisher2.boulder:9091 health check
boulder-boulder-1 | 2022-10-27T12:16:25.446688+00:00 263e8a95e41c unknown health-checker[413]: 3 health-checker r_m5yww [AUDIT] timed out waiting for va1.boulder:9097 health check
boulder-boulder-1 | 2022-10-27T12:16:53.633389+00:00 263e8a95e41c unknown health-checker[412]: 3 health-checker kYynsQE [AUDIT] timed out waiting for nonce2.boulder:9101 health check
boulder-boulder-1 | 2022-10-27T12:17:22.62244+00:00 263e8a95e41c unknown health-checker[388]: 3 health-checker r_m5yww [AUDIT] timed out waiting for va1.boulder:9097 health check
boulder-boulder-1 | 2022-10-27T12:17:51.278415+00:00 263e8a95e41c unknown health-checker[382]: 3 health-checker 2ZDr_wk [AUDIT] timed out waiting for va1.boulder:9098 health check
I have labca running on a proxmox VM with Debian 11 and nothing else on it. Since this error appears my NUC draws a lot more power, just this VM adding about 10W where the whole system with 5LXC and 3 other VMs usually uses 10W.
Please let me know if you need any more information.
from labca.
ChrisThePCGeek
commented on August 10, 2024
I have this issue also on a VM with debian 11 installed. no other services on it.
these two show in the dashboard repeating while the boulder container continues to crash loop
[AUDIT] timed out waiting for sa1.boulder:9095 health check13 seconds
[AUDIT] timed out waiting for nonce2.boulder:9101 health check
thought it was just me. thankfully not but now all broken. I restored my vm from a backup and it failed to work also for some other error regarding an ip of 10.88.88.88 (assuming the version of boulder is too old from that vm or something) I havent updated it since install.
from labca.
NateTheSage
commented on August 10, 2024
Glad to see I'm not alone!
I've already got...something on the order of like a hundred certificates already issued with automated processes. Reinstalling isn't really an option for me, but I'm pleased at least to see it's not just me.
No other container is causing issues still, fortunately.
Although I am getting tired of my certcheck emailing me every hour for a certificate that needs renewed. 🤣
from labca.
ChrisThePCGeek
commented on August 10, 2024
Glad to see I'm not alone!
I've already got...something on the order of like a hundred certificates already issued with automated processes. Reinstalling isn't really an option for me, but I'm pleased at least to see it's not just me.
No other container is causing issues still, fortunately.
Although I am getting tired of my certcheck emailing me every hour for a certificate that needs renewed. 🤣
I tried re-installing and restored one of the weekly backups but still same issue.
from labca.
ChrisThePCGeek
commented on August 10, 2024
I installed a fresh run in a new debian 11 vm. all went well until restoring my backup data .tgz file I downloaded from my old instance and it went to perform a restart. then got this in the boulder container logs at the end when it crashed:
01 health check
Starting service sd-test-srv
Starting service nonce-service-taro
Error starting service nonce-service-taro: Command '['./bin/health-checker', '-addr', 'nonce1.boulder:9101', '-config', 'labca/config/health-checker.json']' returned non-zero exit status 1.
from labca.
NateTheSage
commented on August 10, 2024
Interesting. Well, glad to know trying to redo this whole shebang would have resulted in the same.
Did the service change every once and a while? It's sometimes the same service, then sometimes it's not.
from labca.
ChrisThePCGeek
commented on August 10, 2024
it does change, I think each restart.
from labca.
NateTheSage
commented on August 10, 2024
That would be what I've observed too. Sometimes it's the same service, more often than not it's something different.
from labca.
hakwerk
commented on August 10, 2024
There was indeed an issue in generating the config files, resulting in the constant restart loop.
I created a new release (v22.10.2) and in my testing on several machines the issue no longer occurs. Please update to the latest version and hopefully your LabCA installs will run stably again
from labca.
ChrisThePCGeek
commented on August 10, 2024
Thank you!! I re-ran the install to update and its working fine now. Much appreciated.
from labca.
NateTheSage
commented on August 10, 2024
Can confirm, everything seems peachy now. I'm going to see if I can force run my update process and make sure all's well.
EDIT: Standing corrected, I'm continually restarting again. One sec while I get the logs and see if just doing a docker-compose pull helps.
SUPER EDIT: Standing further corrected! It seems I just needed to reboot the container, and I stopped getting my 502. I'm going to make sure at least one of my services can pull its new cert.
from labca.
NateTheSage
commented on August 10, 2024
# INFO: Using main config file /etc/dehydrated/config
# INFO: Using additional config file /etc/dehydrated/conf.d/hook.sh
# INFO: Running /usr/bin/dehydrated as www-data/www-data
# INFO: Using main config file /etc/dehydrated/config
# INFO: Using additional config file /etc/dehydrated/conf.d/hook.sh
+ Hook: Nothing to do...
+ Hook: Nothing to do...
Processing bookmarks.mydomain.net
+ Hook: Nothing to do...
+ Checking domain name(s) of existing cert... unchanged.
+ Checking expire date of existing cert...
+ Valid till Nov 26 05:00:15 2022 GMT (Less than 30 days). Renewing!
+ Signing domains...
+ Generating private key...
+ Generating signing request...
+ Requesting new certificate order from CA...
+ Received 1 authorizations URLs from the CA
+ Handling authorization for bookmarks.mydomain.net
+ 1 pending challenge(s)
+ Deploying challenge tokens...
+ Hook: Nothing to do...
+ Responding to challenge for bookmarks.mydomain.net authorization...
+ Challenge is valid!
+ Cleaning challenge tokens...
+ Hook: Nothing to do...
+ Requesting certificate...
+ Checking certificate...
+ Done!
+ Creating fullchain.pem...
+ Hook: Nothing to do...
+ Hook: restarting Apache for bookmarks.mydomain.net...
Restarting apache2 (via systemctl): apache2.service.
+ Done!
+ Hook: Nothing to do...
Flawless! Everyone who tried to renew their cert on the hour got it renewed. Thanks for your assistance, @hakwerk
from labca.
NateTheSage
commented on August 10, 2024
Seems it started happening again, although this time it's a little more consistent, just ca2.service.consul:9093 and ca1.service.consul:9093.
@hakwerk Any chance this is the same issue? I can drop another log if needed, but it otherwise looks like the one I dropped when I first opened this.
Edit: Not as consistent as I'd hoped, this last time it was publisher2.service.consul, and another was rva1.service.consul. I'm going to try updating again to see if that helps, but I think I did that already.
from labca.
hakwerk
commented on August 10, 2024
So it had been working and then suddenly started restarting? Then it should be something else. I haven't had time yet to create a new release so nothing has changed in the code.
Are all other containers running correctly?
from labca.
NateTheSage
commented on August 10, 2024
All other containers running correctly, yeah. It's kind of odd, after the update it was fine, but it just started happening again recently. The only way I noticed was because my certificate updating processes started dropping tons of emails saying the gateway was 502ing again, and I haven't touched or otherwise done much with it. I think the most that's happened is the underlying VM rebooted once for updates.
If you need some additional logging, I'd be happy to provide, or at least open a new issue.
from labca.
hakwerk
commented on August 10, 2024
I have no way of reproducing this issue, so I would need the logs from around the time it transitions from working to not working. Hopefully there is something in the logs then to explain what triggers it
from labca.
NateTheSage
commented on August 10, 2024
nginx.log
labca.log
boulder.log
control.log
commander.log
acme_tiny.log
Here's my latest logs for today. I haven't rebooted the underlying VM in a while, going to see if I can get a fresh install going on the same VM and see what happens.
from labca.
NateTheSage
commented on August 10, 2024
I just did a fresh install, still the same problem.
* Starting enhanced syslogd rsyslogd
...done.
Connected to boulder-mysql:3306
boulder_sa_test
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/boulder_sa.sql
boulder_sa_integration
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/boulder_sa.sql
incidents_sa_test
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/incidents_sa.sql
incidents_sa_integration
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/incidents_sa.sql
database setup complete
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
* Starting enhanced syslogd rsyslogd
...done.
Connected to boulder-mysql:3306
boulder_sa_test
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/boulder_sa.sql
boulder_sa_integration
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/boulder_sa.sql
incidents_sa_test
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/incidents_sa.sql
incidents_sa_integration
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/incidents_sa.sql
database setup complete
Found slot 1775220161 with matching token label.
The key pair has been imported.
Found slot 1305390123 with matching token label.
The key pair has been imported.
echo bin/nonce-service bin/boulder bin/boulder-ca bin/log-validator bin/mail-tester bin/boulder-sa bin/boulder-wfe2 bin/boulder-ra bin/crl-checker bin/notify-mailer bin/orphan-finder bin/boulder-va bin/admin-revoker bin/boulder-publisher bin/id-exporter bin/caa-log-checker bin/ocsp-responder bin/bad-key-revoker bin/ocsp-updater bin/contact-auditor bin/reversed-hostname-checker bin/crl-storer bin/akamai-purger bin/cert-checker bin/expiration-mailer bin/ceremony bin/boulder-observer bin/crl-updater bin/rocsp-tool
bin/nonce-service bin/boulder bin/boulder-ca bin/log-validator bin/mail-tester bin/boulder-sa bin/boulder-wfe2 bin/boulder-ra bin/crl-checker bin/notify-mailer bin/orphan-finder bin/boulder-va bin/admin-revoker bin/boulder-publisher bin/id-exporter bin/caa-log-checker bin/ocsp-responder bin/bad-key-revoker bin/ocsp-updater bin/contact-auditor bin/reversed-hostname-checker bin/crl-storer bin/akamai-purger bin/cert-checker bin/expiration-mailer bin/ceremony bin/boulder-observer bin/crl-updater bin/rocsp-tool
GOBIN=/boulder/bin GO111MODULE=on go install -mod=vendor -buildvcs=false -tags "integration" ./...
root@fukase:/home/labca# docker logs boulder-boulder-1 -f
* Starting enhanced syslogd rsyslogd
...done.
Connected to boulder-mysql:3306
boulder_sa_test
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/boulder_sa.sql
boulder_sa_integration
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/boulder_sa.sql
incidents_sa_test
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/incidents_sa.sql
incidents_sa_integration
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/incidents_sa.sql
database setup complete
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
Waiting for /boulder/labca/setup_complete to appear...
* Starting enhanced syslogd rsyslogd
...done.
Connected to boulder-mysql:3306
boulder_sa_test
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/boulder_sa.sql
boulder_sa_integration
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/boulder_sa.sql
incidents_sa_test
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/incidents_sa.sql
incidents_sa_integration
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/incidents_sa.sql
database setup complete
Found slot 1775220161 with matching token label.
The key pair has been imported.
Found slot 1305390123 with matching token label.
The key pair has been imported.
echo bin/nonce-service bin/boulder bin/boulder-ca bin/log-validator bin/mail-tester bin/boulder-sa bin/boulder-wfe2 bin/boulder-ra bin/crl-checker bin/notify-mailer bin/orphan-finder bin/boulder-va bin/admin-revoker bin/boulder-publisher bin/id-exporter bin/caa-log-checker bin/ocsp-responder bin/bad-key-revoker bin/ocsp-updater bin/contact-auditor bin/reversed-hostname-checker bin/crl-storer bin/akamai-purger bin/cert-checker bin/expiration-mailer bin/ceremony bin/boulder-observer bin/crl-updater bin/rocsp-tool
bin/nonce-service bin/boulder bin/boulder-ca bin/log-validator bin/mail-tester bin/boulder-sa bin/boulder-wfe2 bin/boulder-ra bin/crl-checker bin/notify-mailer bin/orphan-finder bin/boulder-va bin/admin-revoker bin/boulder-publisher bin/id-exporter bin/caa-log-checker bin/ocsp-responder bin/bad-key-revoker bin/ocsp-updater bin/contact-auditor bin/reversed-hostname-checker bin/crl-storer bin/akamai-purger bin/cert-checker bin/expiration-mailer bin/ceremony bin/boulder-observer bin/crl-updater bin/rocsp-tool
GOBIN=/boulder/bin GO111MODULE=on go install -mod=vendor -buildvcs=false -tags "integration" ./...
* Starting enhanced syslogd rsyslogd
...done.
Connected to boulder-mysql:3306
boulder_sa_test
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/boulder_sa.sql
boulder_sa_integration
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/boulder_sa.sql
incidents_sa_test
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/incidents_sa.sql
incidents_sa_integration
Already exists - skipping create
Applied 0 migrations
Added users from ../db-users/incidents_sa.sql
database setup complete
CKR_SLOT_ID_INVALID: Slot 0 does not exist.
Found slot 1775220161 with matching token label.
The key pair has been imported.
CKR_SLOT_ID_INVALID: Slot 1 does not exist.
Found slot 1305390123 with matching token label.
The key pair has been imported.
echo bin/nonce-service bin/boulder bin/boulder-ca bin/log-validator bin/mail-tester bin/boulder-sa bin/boulder-wfe2 bin/boulder-ra bin/crl-checker bin/notify-mailer bin/orphan-finder bin/boulder-va bin/admin-revoker bin/boulder-publisher bin/id-exporter bin/caa-log-checker bin/ocsp-responder bin/bad-key-revoker bin/ocsp-updater bin/contact-auditor bin/reversed-hostname-checker bin/crl-storer bin/akamai-purger bin/cert-checker bin/expiration-mailer bin/ceremony bin/boulder-observer bin/crl-updater bin/rocsp-tool
bin/nonce-service bin/boulder bin/boulder-ca bin/log-validator bin/mail-tester bin/boulder-sa bin/boulder-wfe2 bin/boulder-ra bin/crl-checker bin/notify-mailer bin/orphan-finder bin/boulder-va bin/admin-revoker bin/boulder-publisher bin/id-exporter bin/caa-log-checker bin/ocsp-responder bin/bad-key-revoker bin/ocsp-updater bin/contact-auditor bin/reversed-hostname-checker bin/crl-storer bin/akamai-purger bin/cert-checker bin/expiration-mailer bin/ceremony bin/boulder-observer bin/crl-updater bin/rocsp-tool
GOBIN=/boulder/bin GO111MODULE=on go install -mod=vendor -buildvcs=false -tags "integration" ./...
./link.sh
pebble-challtestsrv - 2022/12/01 20:16:16 Creating HTTP-01 challenge server on 10.77.77.77:5002
pebble-challtestsrv - 2022/12/01 20:16:16 Creating HTTPS HTTP-01 challenge server on 10.77.77.77:5001
pebble-challtestsrv - 2022/12/01 20:16:16 Creating TCP and UDP DNS-01 challenge server on :8053
pebble-challtestsrv - 2022/12/01 20:16:16 Creating TCP and UDP DNS-01 challenge server on :8054
pebble-challtestsrv - 2022/12/01 20:16:16 Creating TLS-ALPN-01 challenge server on 10.88.88.88:5001
pebble-challtestsrv - 2022/12/01 20:16:16 Answering A queries with 10.77.77.77 by default
pebble-challtestsrv - 2022/12/01 20:16:16 Starting challenge servers
pebble-challtestsrv - 2022/12/01 20:16:16 Starting management server on :8055
2022-12-01T20:16:16.704746+00:00Z 8272327868c5 mail-test-srv[4685]: 6 mail-test-srv gdmZ2gI mail-test-srv: Got connection from 127.0.0.1:47110
2022-12-01T20:16:16.706790+00:00Z 8272327868c5 mail-test-srv[4685]: 6 mail-test-srv pILs9wY 2022/12/01 20:16:16 mail-test-srv: 127.0.0.1:47110: readline: EOF
2022-12-01T20:16:16.860522+00:00Z 8272327868c5 boulder-remoteva[4707]: 6 boulder-remoteva hZuhGAA Versions: boulder-remoteva=(Unspecified Unspecified) Golang=(go1.19.2) BuildHost=(Unspecified)
Connecting to rva1.service.consul:9098 health service
2022-12-01T20:16:17.370877+00:00Z 8272327868c5 boulder-publisher[4718]: 6 boulder-publisher v7vfswY Versions: boulder-publisher=(Unspecified Unspecified) Golang=(go1.19.2) BuildHost=(Unspecified)
Connecting to publisher2.service.consul:9091 health service
2022-12-01T20:16:17.615349+00:00Z 8272327868c5 log-validator[4739]: 6 log-validator 1c3GjAk Waiting for /var/log/akamai-purger.log to appear...
2022-12-01T20:16:17.616919+00:00Z 8272327868c5 log-validator[4739]: 6 log-validator rsLnrwg Waiting for /var/log/bad-key-revoker.log to appear...
2022-12-01T20:16:17.617797+00:00Z 8272327868c5 log-validator[4739]: 6 log-validator ucH95gQ Waiting for /var/log/boulder-ca.log to appear...
2022-12-01T20:16:17.618644+00:00Z 8272327868c5 log-validator[4739]: 6 log-validator xdaeiQk Waiting for /var/log/boulder-observer.log to appear...
2022-12-01T20:16:17.619683+00:00Z 8272327868c5 log-validator[4739]: 6 log-validator 67f_wwg Waiting for /var/log/boulder-ra.log to appear...
2022-12-01T20:16:17.620486+00:00Z 8272327868c5 log-validator[4739]: 6 log-validator _uqR4wM Waiting for /var/log/boulder-remoteva.log to appear...
2022-12-01T20:16:17.621331+00:00Z 8272327868c5 log-validator[4739]: 6 log-validator s7f1_AU Waiting for /var/log/boulder-sa.log to appear...
2022-12-01T20:16:17.622147+00:00Z 8272327868c5 log-validator[4739]: 6 log-validator iazfiAY Waiting for /var/log/boulder-va.log to appear...
2022-12-01T20:16:17.623010+00:00Z 8272327868c5 log-validator[4739]: 6 log-validator g5CP7Ac Waiting for /var/log/boulder-wfe2.log to appear...
2022-12-01T20:16:17.623904+00:00Z 8272327868c5 log-validator[4739]: 6 log-validator l7fb5Qs Waiting for /var/log/crl-storer.log to appear...
2022-12-01T20:16:17.624709+00:00Z 8272327868c5 log-validator[4739]: 6 log-validator 8uCB5gQ Waiting for /var/log/crl-updater.log to appear...
2022-12-01T20:16:17.625542+00:00Z 8272327868c5 log-validator[4739]: 6 log-validator id2G3Aw Waiting for /var/log/nonce-service.log to appear...
2022-12-01T20:16:17.626377+00:00Z 8272327868c5 log-validator[4739]: 6 log-validator x8_tqgs Waiting for /var/log/ocsp-responder.log to appear...
2022-12-01T20:16:17.627333+00:00Z 8272327868c5 log-validator[4739]: 6 log-validator 5fDH5gY Waiting for /var/log/ocsp-updater.log to appear...
2022-12-01T20:16:17.771082+00:00Z 8272327868c5 boulder-remoteva[4745]: 6 boulder-remoteva hZuhGAA Versions: boulder-remoteva=(Unspecified Unspecified) Golang=(go1.19.2) BuildHost=(Unspecified)
Connecting to rva1.service.consul:9097 health service
2022-12-01T20:16:18.171910+00:00Z 8272327868c5 boulder-publisher[4756]: 6 boulder-publisher v7vfswY Versions: boulder-publisher=(Unspecified Unspecified) Golang=(go1.19.2) BuildHost=(Unspecified)
Connecting to publisher1.service.consul:9091 health service
2022-12-01T20:16:18.377863+00:00Z 8272327868c5 boulder-va[4776]: 6 boulder-va wqOb_gM Versions: boulder-va=(Unspecified Unspecified) Golang=(go1.19.2) BuildHost=(Unspecified)
Connecting to va2.service.consul:9092 health service
2022-12-01T20:16:18.546211+00:00Z 8272327868c5 nonce-service[4793]: 6 nonce-service wva65gI Versions: nonce-service=(Unspecified Unspecified) Golang=(go1.19.2) BuildHost=(Unspecified)
Connecting to nonce1.service.consul:9101 health service
2022-12-01T20:16:18.791398+00:00Z 8272327868c5 nonce-service[4804]: 6 nonce-service wva65gI Versions: nonce-service=(Unspecified Unspecified) Golang=(go1.19.2) BuildHost=(Unspecified)
Connecting to nonce2.service.consul:9101 health service
2022/12/01 20:16:18 ct-test-srv on :4500 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEYggOxPnPkzKBIhTacSYoIfnSL2jPugcbUKx83vFMvk5gKAz/AGe87w20riuPwEGn229hKVbEKHFB61NIqNHC3Q==
2022/12/01 20:16:18 ct-test-srv on :4501 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEKtnFevaXV/kB8dmhCNZHmxKVLcHX1plaAsY9LrKilhYxdmQZiu36LvAvosTsqMVqRK9a96nC8VaxAdaHUbM8EA==
2022/12/01 20:16:18 ct-test-srv on :4510 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEyw1HymhJkuxSIgt3gqW3sVXqMqB3EFsXcMfPFo0vYwjNiRmCJDXKsR0Flp7MAK+wc3X/7Hpc8liUbMhPet7tEA==
2022/12/01 20:16:18 ct-test-srv on :4511 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEFRu37ZRLg8lT4rVQwMwh4oAOpXb4Sx+9hgQ+JFCjmAv3oDV+sDOMsC7hULkGTn+LB5L1SRo/XIY4Kw5V+nFXgg==
2022/12/01 20:16:18 ct-test-srv on :4512 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEFRu37ZRLg8lT4rVQwMwh4oAOpXb4Sx+9hgQ+JFCjmAv3oDV+sDOMsC7hULkGTn+LB5L1SRo/XIY4Kw5V+nFXgg==
2022/12/01 20:16:18 ct-test-srv on :4600 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAExhriVaEwBOtdNzg5EOtJBHl/u+ua1FtCR/CBXQ1kvpFelcP3gozLNXyxV/UexuifpmzTN31CdfdHv1kK3KDIxQ==
2022/12/01 20:16:18 ct-test-srv on :4601 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE7uzW0zXQpWIk7MZUBdTu1muNzekMCIv/kn16+ifndQ584DElobOJ0ZlcACz9WdFyGTjOCfAqBmFybX2OJKfFVg==
2022/12/01 20:16:18 ct-test-srv on :4602 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE/s5W5OHfowdLA7KerJ+mOizfHJE6Snfib8ueoBYl8Y12lpOoJTtCmmrx4m9KAb9AptInWpGrIaLY+5Y29l2eGw==
2022/12/01 20:16:18 ct-test-srv on :4603 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE2EFdA2UBfbJ2Sw1413hBN9YESyABmTGbdgcMh0l/GyV3eFrFjcVS0laNphkfRZ+qkcMbeF+IIHqVzxHAM/2mQQ==
2022/12/01 20:16:18 ct-test-srv on :4604 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEAMSHwrzvr/KvNmUT55+uQo7CXQLPx1X+qEdKGekUg1q/InN/E37bCY/x45wC00qgiE0D3xoxnUJbKaCQcAX39w==
2022/12/01 20:16:18 ct-test-srv on :4605 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEzmpksKS/mHgJZ821po3ldwonsz3K19jwsZgNSGYvEuzAVtWbGfY+6aUXua7f8WK8l2amHETISOY4JTRwk5QFyw==
2022/12/01 20:16:18 ct-test-srv on :4606 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE31BxBVCdehxOC35jJzvAPNrU4ZjNXbmxS+zSN5DSkpJWQUp5wUHPGnXiSCtx7jXnTYLVzslIyXWpNN8m8BiKjQ==
2022/12/01 20:16:18 ct-test-srv on :4607 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEAjRx6Mhc/U4Ye7NzsZ7bbKMGhKVpGZHpZJMzLzNIveBAPh5OBDHpSdn9RY58t4diH8YLjqCi9o+k1T5RwiFbfQ==
2022/12/01 20:16:18 ct-test-srv on :4608 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEsHFSkgrlrwIY0PG79tOZhPvBzrnrpbrWa3pG2FfkLeEJQ2Uvgw1oTZZ+oXcrm4Yb3khWDbpkzDbupI+e8xloeA==
2022/12/01 20:16:18 ct-test-srv on :4609 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEMVjHUOxzh2flagPhuEYy/AhAlpD9qqACg4fGcCxOhLU35r21CQXzKDdCHMu69QDFd6EAe8iGFsybg+Yn4/njtA==
2022-12-01T20:16:19.054108+00:00Z 8272327868c5 boulder-sa[4825]: 6 boulder-sa xJykwg0 Versions: boulder-sa=(Unspecified Unspecified) Golang=(go1.19.2) BuildHost=(Unspecified)
Connecting to sa2.service.consul:9095 health service
got error connecting to health service sa2.service.consul:9095: grpc.health.v1.Health.Check timed out after 1001 ms
Connecting to sa2.service.consul:9095 health service
2022-12-01T20:16:20.215672+00:00Z 8272327868c5 boulder-sa[4853]: 6 boulder-sa xJykwg0 Versions: boulder-sa=(Unspecified Unspecified) Golang=(go1.19.2) BuildHost=(Unspecified)
Connecting to sa1.service.consul:9095 health service
2022-12-01T20:16:20.508640+00:00Z 8272327868c5 boulder-va[4865]: 6 boulder-va wqOb_gM Versions: boulder-va=(Unspecified Unspecified) Golang=(go1.19.2) BuildHost=(Unspecified)
Connecting to va1.service.consul:9092 health service
2022-12-01T20:16:22.128181+00:00Z 8272327868c5 crl-storer[4877]: 6 crl-storer v5rF8w4 Versions: crl-storer=(Unspecified Unspecified) Golang=(go1.19.2) BuildHost=(Unspecified)
2022-12-01T20:16:22.199185+00:00Z 8272327868c5 boulder-ca[4899]: 6 boulder-ca s7b28wQ Versions: boulder-ca=(Unspecified Unspecified) Golang=(go1.19.2) BuildHost=(Unspecified)
2022-12-01T20:16:22.200721+00:00Z 8272327868c5 boulder-ca[4899]: 6 boulder-ca o73C-w4 loading hostname policy, sha256: 5d74cbde75acdae9c60f2633937f488a2740d3e5e2166f08370481230c659cb4
Connecting to ca2.service.consul:9093 health service
got error connecting to health service ca2.service.consul:9093: grpc.health.v1.Health.Check timed out after 1003 ms
Connecting to ca2.service.consul:9093 health service
got error connecting to health service ca2.service.consul:9093: grpc.health.v1.Health.Check timed out after 1000 ms
Connecting to ca2.service.consul:9093 health service
got error connecting to health service ca2.service.consul:9093: grpc.health.v1.Health.Check timed out after 1000 ms
Connecting to ca2.service.consul:9093 health service
got error connecting to health service ca2.service.consul:9093: grpc.health.v1.Health.Check timed out after 1001 ms
Connecting to ca2.service.consul:9093 health service
got error connecting to health service ca2.service.consul:9093: grpc.health.v1.Health.Check timed out after 1000 ms
Connecting to ca2.service.consul:9093 health service
got error connecting to health service ca2.service.consul:9093: grpc.health.v1.Health.Check timed out after 1000 ms
Connecting to ca2.service.consul:9093 health service
got error connecting to health service ca2.service.consul:9093: grpc.health.v1.Health.Check timed out after 1000 ms
Connecting to ca2.service.consul:9093 health service
got error connecting to health service ca2.service.consul:9093: grpc.health.v1.Health.Check timed out after 1000 ms
Connecting to ca2.service.consul:9093 health service
got error connecting to health service ca2.service.consul:9093: grpc.health.v1.Health.Check timed out after 1000 ms
Connecting to ca2.service.consul:9093 health service
got error connecting to health service ca2.service.consul:9093: grpc.health.v1.Health.Check timed out after 873 ms
2022-12-01T20:16:32.176911+00:00Z 8272327868c5 health-checker[4900]: 3 health-checker 7oaplgQ [AUDIT] timed out waiting for ca2.service.consul:9093 health check
Waiting for debug port 8055 (pebble-challtestsrv --defaultIPv4 10.77.77.77 -defaultIPv6 --dns01 :8053,:8054 --management :8055 --http01 10.77.77.77:5002 -https01 10.77.77.77:5001 --tlsalpn01 10.88.88.88:5001)
Waiting for debug port 8055 (pebble-challtestsrv --defaultIPv4 10.77.77.77 -defaultIPv6 --dns01 :8053,:8054 --management :8055 --http01 10.77.77.77:5002 -https01 10.77.77.77:5001 --tlsalpn01 10.88.88.88:5001)
Starting service mail-test-srv
Waiting for debug port 9380 (./bin/mail-test-srv --closeFirst 5 --cert labca/mail-test-srv/localhost/cert.pem --key labca/mail-test-srv/localhost/key.pem)
Starting service s3-test-srv
Starting service boulder-remoteva-b
Starting service boulder-publisher-2
Starting service akamai-test-srv
Starting service log-validator
Starting service boulder-remoteva-a
Starting service boulder-publisher-1
Starting service boulder-va-2
Starting service nonce-service-taro
Starting service nonce-service-zinc
Starting service ct-test-srv
Starting service boulder-sa-2
Starting service boulder-sa-1
Starting service boulder-va-1
Starting service crl-storer
Waiting for debug port 9667 (./bin/boulder crl-storer --config labca/config/crl-storer.json)
Waiting for debug port 9667 (./bin/boulder crl-storer --config labca/config/crl-storer.json)
Waiting for debug port 9667 (./bin/boulder crl-storer --config labca/config/crl-storer.json)
Waiting for debug port 9667 (./bin/boulder crl-storer --config labca/config/crl-storer.json)
Waiting for debug port 9667 (./bin/boulder crl-storer --config labca/config/crl-storer.json)
Waiting for debug port 9667 (./bin/boulder crl-storer --config labca/config/crl-storer.json)
Waiting for debug port 9667 (./bin/boulder crl-storer --config labca/config/crl-storer.json)
Waiting for debug port 9667 (./bin/boulder crl-storer --config labca/config/crl-storer.json)
Waiting for debug port 9667 (./bin/boulder crl-storer --config labca/config/crl-storer.json)
Waiting for debug port 9667 (./bin/boulder crl-storer --config labca/config/crl-storer.json)
Waiting for debug port 9667 (./bin/boulder crl-storer --config labca/config/crl-storer.json)
Waiting for debug port 9667 (./bin/boulder crl-storer --config labca/config/crl-storer.json)
Waiting for debug port 9667 (./bin/boulder crl-storer --config labca/config/crl-storer.json)
Waiting for debug port 9667 (./bin/boulder crl-storer --config labca/config/crl-storer.json)
Starting service boulder-ca-b
Error starting service boulder-ca-b: Command '['./bin/health-checker', '-addr', 'ca2.service.consul:9093', '-config', 'labca/config/health-checker.json']' returned non-zero exit status 1.
EDIT: Wait a minute, I didn't clear my databases. Let me try this again.
from labca.
NateTheSage
commented on August 10, 2024
...Huh. I don't know what happened, but after clearing my databases and reinstalling from scratch, it works just fine. The timeout takes a bit but it does eventually connect.
I've minorly inconvenienced myself by completely clearing out the databases, but since it works now, with any manner of luck it should stay working. Won't take too long to get it back up to working state anyway. 😄
from labca.
Related Issues (20)
- Error renewing certificate HOT 2
- DNS Domain , CAA Record - Guidance HOT 3
- Error when testing email HOT 4
- reduce clone time with --depth option in installer. HOT 1
- [renew] - Invalid hostname in redirect target, must end in IANA registered TLD HOT 2
- [Feature Request] Helm chart HOT 1
- Error in install last step when Signing certificate... HOT 1
- [DOCS] Probable improvement of the docs on probable errors with a specific use case HOT 2
- Rate limit hit after two requests HOT 1
- Permissions issue with boulder HOT 14
- non tld domains work? HOT 8
- [Feature Request] LabCA standalone Arm container image
- Cannot finalize the installation process - Response Code: 308 HOT 3
- Boulder container restarting - panic: opening hierarchy directory HOT 3
- `labca/entrypoint.sh` not found in `labca-boulder` provided from `ghcr.io/hakwerk/labca-boulder` HOT 1
- Add example on how to use LabCA with certbot HOT 2
- Revoking a certificate does not work HOT 6
- how to stop labca with root or other user HOT 1
- reverse proxy - Traefik HOT 2
- Possibility of creating multiple users for login
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from labca.