letsencrypt / ct-woodpecker Goto Github PK

A tool to monitor a certificate transparency log for operational problems

License: Mozilla Public License 2.0

Go 99.01% Shell 0.28% Dockerfile 0.71%

certificate-transparency prometheus ct x509 pki monitoring ct-woodpecker grafana alert-manager

ct-woodpecker's Introduction

Certbot is part of EFF’s effort to encrypt the entire Internet. Secure communication over the Web relies on HTTPS, which requires the use of a digital certificate that lets browsers verify the identity of web servers (e.g., is that really google.com?). Web servers obtain their certificates from trusted third parties called certificate authorities (CAs). Certbot is an easy-to-use client that fetches a certificate from Let’s Encrypt—an open certificate authority launched by the EFF, Mozilla, and others—and deploys it to a web server.

Anyone who has gone through the trouble of setting up a secure website knows what a hassle getting and maintaining a certificate is. Certbot and Let’s Encrypt can automate away the pain and let you turn on and manage HTTPS with simple commands. Using Certbot and Let's Encrypt is free.

Getting Started

The best way to get started is to use our interactive guide. It generates instructions based on your configuration settings. In most cases, you’ll need root or administrator access to your web server to run Certbot.

Certbot is meant to be run directly on your web server on the command line, not on your personal computer. If you’re using a hosted service and don’t have direct access to your web server, you might not be able to use Certbot. Check with your hosting provider for documentation about uploading certificates or using certificates issued by Let’s Encrypt.

Contributing

If you'd like to contribute to this project please read Developer Guide.

This project is governed by EFF's Public Projects Code of Conduct.

Links

Documentation: https://certbot.eff.org/docs

Software project: https://github.com/certbot/certbot

Changelog: https://github.com/certbot/certbot/blob/master/certbot/CHANGELOG.md

For Contributors: https://certbot.eff.org/docs/contributing.html

For Users: https://certbot.eff.org/docs/using.html

Main Website: https://certbot.eff.org

Let's Encrypt Website: https://letsencrypt.org

Community: https://community.letsencrypt.org

ACME spec: RFC 8555

ACME working area in github (archived): https://github.com/ietf-wg-acme/acme

Current Features

Supports multiple web servers:
- Apache 2.4+
- nginx/0.8.48+
- webroot (adds files to webroot directories in order to prove control of domains and obtain certificates)
- standalone (runs its own simple webserver to prove you control a domain)
- other server software via third party plugins
The private key is generated locally on your system.
Can talk to the Let's Encrypt CA or optionally to other ACME compliant services.
Can get domain-validated (DV) certificates.
Can revoke certificates.
Supports ECDSA (default) and RSA certificate private keys.
Can optionally install a http -> https redirect, so your site effectively runs https only.
Fully automated.
Configuration changes are logged and can be reverted.

ct-woodpecker's People

Contributors

Stargazers

Watchers

Forkers

cpu jsha ezekiel pgporada kernt ripleyludicrous1979 allrightwithworld hixio-mh cherrycodey korallin rob-opsi szepeviktor andygabby ksangan81 mcpherrinm k0zero hbcbh1999

ct-woodpecker's Issues

Write Prometheus rules

Add support for gathering pprof data

This would help troubleshoot either a potential memory leak or if the config we're using is the result of the out of memory issues. I'm thinking it's the config. We received this on a server earlier today. Relevant details about the server are

1 vCPU
1.75GB RAM

{
  "metricsAddr": ":1971",
  "dbURI": "<redacted>",
  "fetchConfig": {
    "interval": "120s",
    "timeout": "100s"
  },
  "submitConfig": {
    "interval": "600s",
    "timeout": "500s",
    "certIssuerKeyPath": "<redacted>",
    "certIssuerPath": "<redacted>"
  },
  "inclusionConfig": {
    "interval": "60s",
    "fetchBatchSize": 1000,
    "maxGetEntries": 10000
  },
"logs": [<redacted>]
}

top reported that ct-woodpecker was consuming 60% of available RAM.

Jan 31 19:11:03 [localhost] ct-woodpecker: fatal error: runtime: out of memory
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime stack:
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.throw(0xb65037, 0x16)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/panic.go:608 +0x72
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.sysMap(0xc044000000, 0x4000000, 0x1307ff8)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mem_linux.go:156 +0xc7
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mheap).sysAlloc(0x12ee120, 0x4000000, 0x76220, 0x759bb1)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/malloc.go:619 +0x1c7
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mheap).grow(0x12ee120, 0x1, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mheap.go:920 +0x42
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mheap).allocSpanLocked(0x12ee120, 0x1, 0x1308008, 0xe0003f540f)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mheap.go:848 +0x337
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mheap).alloc_m(0x12ee120, 0x1, 0x3c, 0xc000558600)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mheap.go:692 +0x119
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mheap).alloc.func1()
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mheap.go:759 +0x4c
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mheap).alloc(0x12ee120, 0x1, 0x101003c, 0x4638c1)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mheap.go:758 +0x8a
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mcentral).grow(0x12f0358, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mcentral.go:232 +0x94
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mcentral).cacheSpan(0x12f0358, 0x7f3ecb87c350)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mcentral.go:106 +0x2f8
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mcache).refill(0x7f3ef748a000, 0x7f3ecabaa03c)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mcache.go:122 +0x95
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mcache).nextFree.func1()
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/malloc.go:749 +0x32
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.systemstack(0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:351 +0x66
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.mstart()
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:1229
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 44 [running]:
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.systemstack_switch()
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc0092d53b8 sp=0xc0092d53b0 pc=0x4617e0
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mcache).nextFree(0x7f3ef748a000, 0x3c, 0x0, 0x0, 0x197)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/malloc.go:748 +0xb6 fp=0xc0092d5410 sp=0xc0092d53b8 pc=0x4166f6
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.mallocgc(0x380, 0xad54c0, 0xa2d301, 0xc043ffc820)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/malloc.go:903 +0x793 fp=0xc0092d54b0 sp=0xc0092d5410 pc=0x417043
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.growslice(0xad54c0, 0xc043f83340, 0x8, 0x8, 0x9, 0x2, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/slice.go:204 +0x145 fp=0xc0092d5518 sp=0xc0092d54b0 pc=0x44c8e5
Jan 31 19:11:03 [localhost] systemd: ct-woodpecker.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/x509.parseCertificate(0xc043e82900, 0x541, 0x541, 0xa32120)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/x509/x509.go:1723 +0x3005 fp=0xc0092d58e8 sp=0xc0092d5518 pc=0x751b45
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/x509.ParseCertificate(0xc043edcb00, 0x541, 0x541, 0x60, 0xb09280, 0x1)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/x509/x509.go:1955 +0x126 fp=0xc0092d5970 sp=0xc0092d58e8 pc=0x752156
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go.(*MerkleTreeLeaf).X509Certificate(0xc043fdd188, 0xc043c3b4a0, 0xc0421bcf00, 0x4b5)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/types.go:380 +0xa4 fp=0xc0092d59b8 sp=0xc0092d5970 pc=0x75ba54
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go.(*RawLogEntry).ToLogEntry(0xc043fdd180, 0xc0092d5b78, 0xc043fdd180, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/serialization.go:302 +0x371 fp=0xc0092d5ab0 sp=0xc0092d59b8 pc=0x759bb1
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go.LogEntryFromLeaf(0x2ddb5a6, 0xc0092d5b78, 0xc043c3b400, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/serialization.go:338 +0x73 fp=0xc0092d5ae8 sp=0xc0092d5ab0 pc=0x759d73
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/client.(*LogClient).GetEntries(0xc000031080, 0xc09060, 0xc000034028, 0x2ddb420, 0x2ddb808, 0x3e8, 0x3e8, 0x3e8, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/client/getentries.go:68 +0x19a fp=0xc0092d5be8 sp=0xc0092d5ae8 pc=0x7fed8a
Jan 31 19:11:03 [localhost] systemd: Unit ct-woodpecker.service entered failed state.
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).getEntries(0xc000149180, 0x2ddb420, 0x32c56cd, 0xc025104c80, 0x0, 0x0, 0xc099e0, 0x1305f20, 0xc00008f180)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:208 +0x24d fp=0xc0092d5cd0 sp=0xc0092d5be8 pc=0x9270dd
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).checkInclusion(0xc000149180, 0xc0092d5f58, 0x2)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:171 +0x61a fp=0xc0092d5ee0 sp=0xc0092d5cd0 pc=0x9263da
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc000149180)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:116 +0x109 fp=0xc0092d5fd8 sp=0xc0092d5ee0 pc=0x92af89
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.goexit()
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0092d5fe0 sp=0xc0092d5fd8 pc=0x4638c1
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 1 [chan receive, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/cmd.WaitForSignal(0xc00008f180, 0xc000184f58)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/cmd/common.go:21 +0x15a
Jan 31 19:11:03 [localhost] ct-woodpecker: main.main()
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/cmd/ct-woodpecker/main.go:51 +0x1ed
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 5 [syscall, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: os/signal.signal_recv(0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/sigqueue.go:139 +0x9c
Jan 31 19:11:03 [localhost] ct-woodpecker: os/signal.loop()
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
Jan 31 19:11:03 [localhost] ct-woodpecker: created by os/signal.init.0
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/os/signal/signal_unix.go:29 +0x41
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 8 [chan receive]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x12e7fe0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/golang/glog/glog.go:882 +0x8b
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/vendor/github.com/golang/glog.init.0
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/golang/glog/glog.go:410 +0x203
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 23 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] systemd: ct-woodpecker.service failed.
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc00010c180, 0xc09020, 0xc0000310c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 24 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc00010c180, 0xc09020, 0xc0000310c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 25 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc00010c300, 0xc09020, 0xc000031400)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 26 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc00010c300, 0xc09020, 0xc000031400)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 27 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc00010c480, 0xc09020, 0xc000031700)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 28 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc00010c480, 0xc09020, 0xc000031700)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 29 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc00010c600, 0xc09020, 0xc000031a40)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 30 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc00010c600, 0xc09020, 0xc000031a40)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 31 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc00010c780, 0xc09020, 0xc000031d40)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 32 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc00010c780, 0xc09020, 0xc000031d40)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 33 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc00010c900, 0xc09020, 0xc00029a080)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 34 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc00010c900, 0xc09020, 0xc00029a080)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 35 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc00010ca80, 0xc09020, 0xc00029a380)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 36 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc00010ca80, 0xc09020, 0xc00029a380)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 37 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc00010cc00, 0xc09020, 0xc00029a680)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 38 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc00010cc00, 0xc09020, 0xc00029a680)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 41 [IO wait]:
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.runtime_pollWait(0x7f3ef742ef00, 0x72, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/netpoll.go:173 +0x66
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*pollDesc).wait(0xc00010b618, 0x72, 0xc000030100, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*pollDesc).waitRead(0xc00010b618, 0xffffffffffffff00, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*FD).Accept(0xc00010b600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1a0
Jan 31 19:11:03 [localhost] ct-woodpecker: net.(*netFD).accept(0xc00010b600, 0x416b9f, 0xc00eb900a0, 0xa0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/fd_unix.go:238 +0x42
Jan 31 19:11:03 [localhost] ct-woodpecker: net.(*TCPListener).accept(0xc00000e4a0, 0xc00005fe00, 0xee85a2a9, 0x2383f89cbcc220f4)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/tcpsock_posix.go:139 +0x2e
Jan 31 19:11:03 [localhost] ct-woodpecker: net.(*TCPListener).AcceptTCP(0xc00000e4a0, 0xc00005fe28, 0x4933b6, 0x5c534838)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/tcpsock.go:247 +0x47
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.tcpKeepAliveListener.Accept(0xc00000e4a0, 0xc00005fe78, 0x18, 0xc0001eb800, 0x6bd135)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/server.go:3232 +0x2f
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.(*Server).Serve(0xc0001615f0, 0xc08ba0, 0xc00000e4a0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/server.go:2826 +0x22f
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.(*Server).ListenAndServe(0xc0001615f0, 0x0, 0xc0002a2f90)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/server.go:2764 +0xb6
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/woodpecker.(*Woodpecker).Run.func1(0xc0001f32c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/woodpecker/woodpecker.go:396 +0x33
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/woodpecker.(*Woodpecker).Run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/woodpecker/woodpecker.go:395 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 42 [select, 1 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc000148fa0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 43 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc00010c240)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 45 [select, 1 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc0001492c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 46 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc00010c3c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 47 [select]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc0001494a0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 48 [select, 1 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc0001495e0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 49 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc00010c540)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 50 [select]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc0001497c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 51 [select, 1 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc000149900)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 52 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc00010c6c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 53 [select]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc000149ae0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 54 [select, 1 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc000149c20)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 55 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc00010c840)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 56 [select]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc000149e00)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 57 [select, 1 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc000149f40)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 58 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc00010c9c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 59 [select]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc0002a4140)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 60 [select, 1 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc0002a4280)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 61 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc00010cb40)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 62 [select]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc0002a4460)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 63 [select, 1 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc0002a45a0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 64 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc00010ccc0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 65 [select]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc0002a4780)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 694 [IO wait]:
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.runtime_pollWait(0x7f3ef742e610, 0x72, 0xc000185870)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/netpoll.go:173 +0x66
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*pollDesc).wait(0xc025104818, 0x72, 0xffffffffffffff00, 0xc015a0, 0x12a9f80)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*pollDesc).waitRead(0xc025104818, 0xc035910400, 0x400, 0x400)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*FD).Read(0xc025104800, 0xc035910400, 0x400, 0x400, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_unix.go:169 +0x179
Jan 31 19:11:03 [localhost] ct-woodpecker: net.(*netFD).Read(0xc025104800, 0xc035910400, 0x400, 0x400, 0x413a8b, 0xc00001e000, 0xa9c200)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/fd_unix.go:202 +0x4f
Jan 31 19:11:03 [localhost] ct-woodpecker: net.(*conn).Read(0xc00000e8a0, 0xc035910400, 0x400, 0x400, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/net.go:177 +0x68
Jan 31 19:11:03 [localhost] ct-woodpecker: crypto/tls.(*block).readFromUntil(0xc03a276a50, 0xc001e0, 0xc00000e8a0, 0x5, 0xc00000e8a0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/crypto/tls/conn.go:492 +0x89
Jan 31 19:11:03 [localhost] ct-woodpecker: crypto/tls.(*Conn).readRecord(0xc007adf880, 0xb86c17, 0xc007adf9a0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/crypto/tls/conn.go:593 +0xdd
Jan 31 19:11:03 [localhost] ct-woodpecker: crypto/tls.(*Conn).Read(0xc007adf880, 0xc031db8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/crypto/tls/conn.go:1145 +0xf1
Jan 31 19:11:03 [localhost] ct-woodpecker: bufio.(*Reader).Read(0xc02bc53bc0, 0xc03e0aa578, 0x9, 0x9, 0xc000122000, 0xc000185c58, 0xc000185c70)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/bufio/bufio.go:216 +0x22f
Jan 31 19:11:03 [localhost] ct-woodpecker: io.ReadAtLeast(0xbff700, 0xc02bc53bc0, 0xc03e0aa578, 0x9, 0x9, 0x9, 0xc000185cf8, 0x6a3f2b, 0xc02bc53ce0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/io/io.go:310 +0x88
Jan 31 19:11:03 [localhost] ct-woodpecker: io.ReadFull(0xbff700, 0xc02bc53bc0, 0xc03e0aa578, 0x9, 0x9, 0xc000185d28, 0x6a3e3d, 0xc000185fb8)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/io/io.go:329 +0x58
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.http2readFrameHeader(0xc03e0aa578, 0x9, 0x9, 0xbff700, 0xc02bc53bc0, 0x0, 0x0, 0xc03a277e00, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:1545 +0x7b
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.(*http2Framer).ReadFrame(0xc03e0aa540, 0xc03a277e00, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:1803 +0xa3
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.(*http2clientConnReadLoop).run(0xc000185fb8, 0xb865f8, 0xc0002a3fb8)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:8263 +0x9e
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.(*http2ClientConn).readLoop(0xc0206608c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:8191 +0x76
Jan 31 19:11:03 [localhost] ct-woodpecker: created by net/http.(*http2Transport).newClientConn
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:7264 +0x636
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 194 [IO wait]:
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.runtime_pollWait(0x7f3ef742e880, 0x72, 0xc000325870)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/netpoll.go:173 +0x66
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*pollDesc).wait(0xc000366598, 0x72, 0xffffffffffffff00, 0xc015a0, 0x12a9f80)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*pollDesc).waitRead(0xc000366598, 0xc0004b8000, 0x2000, 0x2000)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*FD).Read(0xc000366580, 0xc0004b8000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_unix.go:169 +0x179
Jan 31 19:11:03 [localhost] ct-woodpecker: net.(*netFD).Read(0xc000366580, 0xc0004b8000, 0x2000, 0x2000, 0x413a8b, 0xc00001e000, 0xa9c200)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/fd_unix.go:202 +0x4f
Jan 31 19:11:03 [localhost] ct-woodpecker: net.(*conn).Read(0xc00000e6b0, 0xc0004b8000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/net.go:177 +0x68
Jan 31 19:11:03 [localhost] ct-woodpecker: crypto/tls.(*block).readFromUntil(0xc0003c4cc0, 0xc001e0, 0xc00000e6b0, 0x5, 0xc00000e6b0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/crypto/tls/conn.go:492 +0x89
Jan 31 19:11:03 [localhost] ct-woodpecker: crypto/tls.(*Conn).readRecord(0xc0001ec000, 0xb86c17, 0xc0001ec120, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/crypto/tls/conn.go:593 +0xdd
Jan 31 19:11:03 [localhost] ct-woodpecker: crypto/tls.(*Conn).Read(0xc0001ec000, 0xc000438000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/crypto/tls/conn.go:1145 +0xf1
Jan 31 19:11:03 [localhost] ct-woodpecker: bufio.(*Reader).Read(0xc0000333e0, 0xc0001f57d8, 0x9, 0x9, 0xc000122000, 0xc000325c58, 0xc000325c70)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/bufio/bufio.go:216 +0x22f
Jan 31 19:11:03 [localhost] ct-woodpecker: io.ReadAtLeast(0xbff700, 0xc0000333e0, 0xc0001f57d8, 0x9, 0x9, 0x9, 0xc000325cf8, 0x6a3f2b, 0xc03c4c9800)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/io/io.go:310 +0x88
Jan 31 19:11:03 [localhost] ct-woodpecker: io.ReadFull(0xbff700, 0xc0000333e0, 0xc0001f57d8, 0x9, 0x9, 0xc000325d28, 0x6a3e3d, 0xc000325fb8)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/io/io.go:329 +0x58
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.http2readFrameHeader(0xc0001f57d8, 0x9, 0x9, 0xbff700, 0xc0000333e0, 0x0, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:1545 +0x7b
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.(*http2Framer).ReadFrame(0xc0001f57a0, 0xc0410525a0, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:1803 +0xa3
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.(*http2clientConnReadLoop).run(0xc000325fb8, 0xb865f8, 0xc0002b0fb8)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:8263 +0x9e
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.(*http2ClientConn).readLoop(0xc0001096c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:8191 +0x76
Jan 31 19:11:03 [localhost] ct-woodpecker: created by net/http.(*http2Transport).newClientConn
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:7264 +0x636

Write unittests for STH freshness check

maximum_merge_delay isn't needed anymore

The only current use of the maximum_merge_delay config field is to print an error if the most recent STH is older than it. However we now have more sophisticated metrics like sth_age, plus alerts on those. I think we should remove maximum_merge_delay to simplify configuration of ct-woodpecker.

Write README

If woodpecker fails to bind metrics port, it will continue rather than crashing

Check STH-to-STH consistency

ct-woodpecker should verify that the log never violates the append-only properties of a CT log.

Automate docker image build & push

ct-woodpecker should use the same approach we use to build & publish Pebble's docker images.

Handle stale STHs

Some logs have caching frontends and we may see STHs slightly out-of-order. We should handle this gracefully.

Fix `make install` to not overwrite existing config

If you have changed the contents of files in /etc/ct-woodpecker it shouldn't overwrite those files with defaults from test/.

Write integration tests using updated ct-test-srv

This will require refactoring the inclusionChecker, the sctFetcher, and the certSubmitter so that we can call the "work" functions directly without having to interact at the abstraction of the woodpecker like the existing integration tests.

Once refactored we should be able to write small integration tests using the ct-test-srv, switching trees & performing leaf integration on-demand using the management API.

Periodically fetch latest STH and export its age as a Prometheus gauge

Alternately, we could export the timestamp of the STH (instead of the timestamp's age) as a gauge, and check that increase(timestamp[1h]) > 0

Wire inclusionChecker into ct-woodpecker

In 6313966 we added an inclusionChecker to the monitor package conditionally based on whether InclusionOpts are provided in the MonitorOptions provided to monitor.New:

ct-woodpecker/monitor/monitor.go

Lines 180 to 205 in 6313966

 if opts.InclusionOpts != nil { 

 keyHash := sha256.Sum256([]byte(opts.LogKey)) 

 pkBytes, err := base64.StdEncoding.DecodeString(opts.LogKey) 

 if err != nil { 

 return nil, err 

 } 

 pk, err := x509.ParsePKIXPublicKey(pkBytes) 

 if err != nil { 

 return nil, err 

 } 

 sv, err := ct.NewSignatureVerifier(pk) 

 if err != nil { 

 return nil, err 

 } 

 m.inclusionChecker = &inclusionChecker{ 

 logger: logger, 

 client: client, 

 clk: clk, 

 logURI: opts.LogURI, 

 interval: opts.InclusionOpts.Interval, 

 db: db, 

 signatureChecker: sv, 

 logID: big.NewInt(0).SetBytes(keyHash[:]).Int64(), 

 batchSize: opts.InclusionOpts.FetchBatchSize, 

 } 

 }

We missed wiring the ability to configure InclusionOpts in the MonitorOptions that woodpecker.New calls for each of the logs in the ct-woodpecker config:

ct-woodpecker/woodpecker/woodpecker.go

Lines 244 to 271 in 6313966

 var monitors []*monitor.Monitor 

 for _, logConf := range c.Logs { 

 opts := monitor.MonitorOptions{ 

 LogURI: logConf.URI, 

 LogKey: logConf.Key, 

 } 

 if c.FetchConfig != nil { 

 opts.FetchOpts = &monitor.FetcherOptions{ 

 Interval: fetchInterval, 

 Timeout: fetchTimeout, 

 } 

 } 

 if c.SubmitConfig != nil { 

 opts.SubmitOpts = &monitor.SubmitterOptions{ 

 Interval: certInterval, 

 Timeout: certTimeout, 

 IssuerCert: issuerCert, 

 IssuerKey: issuerKey, 

 SubmitPreCert: logConf.SubmitPreCert, 

 SubmitCert: logConf.SubmitCert, 

 } 

 } 

 m, err := monitor.New(opts, logger, clk) 

 if err != nil { 

 return nil, err 

 } 

 monitors = append(monitors, m) 

 }

We should fix that so that the command-line tool can configure & use the inclusion checker.

Inclusion checker doesn't exit cleanly in some conditions

Sending ^C to a ct-woodpecker with an inclusion checker config isn't cleanly shutting down - the process seems to hang and the inclusion checker goroutine still appears to be operating.

Revise integration tests to remove reliance on clock time

Right now the ct-woodpecker integration tests use a function called woodpeckerRun to start an instance of a woodpecker.Woodpecker that is allowed to run for a pre-defined amount of time based on the configured submission/sth fetch intervals before being stopped gracefully:

ct-woodpecker/test/integration_test.go

Lines 73 to 82 in 046129d

 // Start the monitoring process 

 woodpecker.Run() 

 // Sleep for the requested amount of time 

 time.Sleep(duration) 

 // Collect metrics from the woodpecker instance while it is still running 

 metricsData := getWoodpeckerMetrics("http://localhost:1971") 

 woodpecker.Stop()

On my local system this is quite reliable & fast. We can set thresholds in the sub-second range and the tests seem solid. In Travis CI things are less nice: It seems like frequently the test driver kills the woodpecker instance after the configured time but the instance hasn't completed its work as expected. Similarly there are times when the test driver kills the woodpecker instance later than expected.

To work around the first case of flakyness (where the woodpecker is killed before all work is done) we have been adding "padding" to the run duration:

ct-woodpecker/test/integration_test.go

Lines 285 to 287 in 046129d

 padding := time.Millisecond * 50 

 duration := submitInterval*time.Duration(iterations) + padding 

 stdout, metricsData, err := woodpeckerRun(config, duration)

That mostly prevents flaky failures, but they do crop up on occasion.

To work around the second case of flakyness (where the woodpecker is killed later than expected and more work has occurred than planned) we consider our expected values as minimums and allow overages without error. That makes our tests less robust than they could be otherwise.

Overall it feels like we should revisit this testing strategy and come up with something not influenced by clock time.

Perhaps a better alternative is for the woodpecker.Woodpecker to count how many STH fetch/cert submission "cycles" it has done and kill the test based on the reported cycles being == desired. Open to suggestions!

windowStart shouldn't set NotBefore

In pki/certs.go IssueTestCertificate(): If windowStart is defined, the NotBefore in issueLeafCert is set to the same value as windowStart. In most use cases we probably don't want to backdate or future date the NotBefore to the beginning of the temporal shard.

Instead we could have issueLeafCert use now or now -1h for NotBefore, but set NotAfter to sometime within the temporal shard window range.

Also, in both windowStart/windowEnd if statements there is an attempt to perform an AddDate() but is not actually used because the AddDate() return value is not assigned back to the earliest/latest vars (NotAfter in issueLeafCert could then have AddDate(0, 0, -1) removed from the latest var as well).

Write auto-deploy script

Periodically submit certificates from a special-purpose root

To test that a log is accepting submissions, ct-woodpecker should accept a private key and root certificate, with which it can sign new certificates once a minute or so and submit them to the log. This would be a special-purpose root, not trusted by any software except the CT log under test.

ct-woodpecker should export latency and error stats from these submissions, and count it as an error if the SCT fails to verify.

Add database driver support for mysql/mariadb

We've been noticing the ct-woodpecker process crashing over the past few days. It was suggested that sqlite support be swapped out or supplemented with mysql/mariadb support.

$ systemctl status ct-woodpecker
● ct-woodpecker.service - ct-woodpecker log monitor
   Loaded: loaded (/usr/lib/systemd/system/ct-woodpecker.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2019-02-01 05:40:38 UTC; 10h ago
  Process: 8976 ExecStart=/opt/ct-woodpecker -config /etc/ct-woodpecker/config.json -logtostderr (code=exited, status=2)
 Main PID: 8976 (code=exited, status=2)

Jan 30 20:24:28 [localhost] ct-woodpecker: runtime: nelems=64 nalloc=17 previous allocCount=15 nfreed=65534
Jan 30 20:24:28 [localhost] ct-woodpecker: fatal error: sweep increased allocation count
Jan 30 20:24:28 [localhost] ct-woodpecker: runtime stack:
Jan 30 20:24:28 [localhost] ct-woodpecker: runtime.throw(0xb6c66b, 0x20)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/runtime/panic.go:608 +0x72
Jan 30 20:24:28 [localhost] ct-woodpecker: runtime.(*mspan).sweep(0x7fb38b9e8df8, 0x43e100, 0xc000000d00)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mgcsweep.go:298 +0x879
Jan 30 20:24:28 [localhost] ct-woodpecker: runtime.sweepone(0x43e957)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mgcsweep.go:113 +0x10f
Jan 30 20:24:28 [localhost] ct-woodpecker: runtime.gosweepone.func1()
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mgcsweep.go:137 +0x2b
Jan 30 20:24:28 [localhost] ct-woodpecker: runtime.systemstack(0x0)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:351 +0x66
Jan 30 20:24:28 [localhost] ct-woodpecker: runtime.mstart()
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:1229
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 3 [running]:
Jan 30 20:24:28 [localhost] ct-woodpecker: runtime.systemstack_switch()
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc00004d780 sp=0xc00004d778 pc=0x4617e0
Jan 30 20:24:28 [localhost] ct-woodpecker: runtime.gosweepone(0x0)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mgcsweep.go:136 +0x52 fp=0xc00004d7b0 sp=0xc00004d780 pc=0x42b1b2
Jan 30 20:24:28 [localhost] ct-woodpecker: runtime.bgsweep(0xc00006c000)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mgcsweep.go:55 +0xa1 fp=0xc00004d7d8 sp=0xc00004d7b0 pc=0x42ad21
Jan 30 20:24:28 [localhost] ct-woodpecker: runtime.goexit()
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00004d7e0 sp=0xc00004d7d8 pc=0x4638c1
Jan 30 20:24:28 [localhost] ct-woodpecker: created by runtime.gcenable
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mgc.go:216 +0x58
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 1 [chan receive, 1250 minutes]:
Jan 30 20:24:28 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/cmd.WaitForSignal(0xc00008f180, 0xc000563f58)
Jan 30 20:24:28 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/cmd/common.go:21 +0x15a
Jan 30 20:24:28 [localhost] ct-woodpecker: main.main()
Jan 30 20:24:28 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/cmd/ct-woodpecker/main.go:51 +0x1ed
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 5 [syscall, 1250 minutes]:
Jan 30 20:24:28 [localhost] ct-woodpecker: os/signal.signal_recv(0x0)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/runtime/sigqueue.go:139 +0x9c
Jan 30 20:24:28 [localhost] ct-woodpecker: os/signal.loop()
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
Jan 30 20:24:28 [localhost] ct-woodpecker: created by os/signal.init.0
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/os/signal/signal_unix.go:29 +0x41
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 8 [chan receive]:
Jan 30 20:24:28 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x12e7fe0)
Jan 30 20:24:28 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/golang/glog/glog.go:882 +0x8b
Jan 30 20:24:28 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/vendor/github.com/golang/glog.init.0
Jan 30 20:24:28 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/golang/glog/glog.go:410 +0x203
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 23 [select, 1250 minutes]:
Jan 30 20:24:28 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc000104180, 0xc09020, 0xc0000310c0)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 30 20:24:28 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 24 [select, 1250 minutes]:
Jan 30 20:24:28 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc000104180, 0xc09020, 0xc0000310c0)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 30 20:24:28 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 25 [select, 1250 minutes]:
Jan 30 20:24:28 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc000104300, 0xc09020, 0xc000031440)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 30 20:24:28 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 26 [select, 1250 minutes]:
Jan 30 20:24:28 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc000104300, 0xc09020, 0xc000031440)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 30 20:24:28 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 27 [select, 1250 minutes]:
Jan 30 20:24:28 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc000104480, 0xc09020, 0xc000031740)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 30 20:24:28 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 28 [select, 1250 minutes]:
Jan 30 20:24:28 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc000104480, 0xc09020, 0xc000031740)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 30 20:24:28 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 29 [select, 1250 minutes]:
Jan 30 20:24:28 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc000104600, 0xc09020, 0xc000031a40)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 30 20:24:28 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 30 [select, 1250 minutes]:
Jan 30 20:24:28 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc000104600, 0xc09020, 0xc000031a40)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 30 20:24:28 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 31 [select, 1250 minutes]:
Jan 30 20:24:28 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc000104780, 0xc09020, 0xc000031d40)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 30 20:24:28 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 32 [select, 1250 minutes]:
Jan 30 20:24:28 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc000104780, 0xc09020, 0xc000031d40)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 30 20:24:28 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 33 [select, 1250 minutes]:
Jan 30 20:24:28 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc000104900, 0xc09020, 0xc000290080)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 30 20:24:28 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 34 [select, 1250 minutes]:
Jan 30 20:24:28 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc000104900, 0xc09020, 0xc000290080)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 30 20:24:28 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 35 [select, 1250 minutes]:
Jan 30 20:24:28 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc000104a80, 0xc09020, 0xc000290380)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 30 20:24:28 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 30 20:24:28 [localhost] ct-woodpecker: goroutine 36 [select, 1250 minutes]:
Jan 30 20:24:28 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc000104a80, 0xc09020, 0xc000290380)
Jan 30 20:24:28 [localhost] ct-woodpecker: /usr/local/go/sr...

Another log from early this morning

Feb  1 05:40:38 [localhost] ct-woodpecker: runtime: pointer 0xc002b3a600 to unallocated span span.base()=0xc002b3a000 span.limit=0xc002b4bdf0 span.state=0
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime: found in object at *(0xc0080483c0+0x20)
Feb  1 05:40:38 [localhost] ct-woodpecker: object=0xc0080483c0 s.base()=0xc008048000 s.limit=0xc008049fe0 s.spanclass=14 s.elemsize=96 s.state=_MSpanInUse
Feb  1 05:40:38 [localhost] ct-woodpecker: *(object+0) = 0x33dff7a
Feb  1 05:40:38 [localhost] ct-woodpecker: *(object+8) = 0x0
Feb  1 05:40:38 [localhost] ct-woodpecker: *(object+16) = 0x0
Feb  1 05:40:38 [localhost] ct-woodpecker: *(object+24) = 0xc003ad56c0
Feb  1 05:40:38 [localhost] ct-woodpecker: *(object+32) = 0xc002b3a600 <==
Feb  1 05:40:38 [localhost] ct-woodpecker: *(object+40) = 0x0
Feb  1 05:40:38 [localhost] ct-woodpecker: *(object+48) = 0x0
Feb  1 05:40:38 [localhost] ct-woodpecker: *(object+56) = 0x0
Feb  1 05:40:38 [localhost] ct-woodpecker: *(object+64) = 0x0
Feb  1 05:40:38 [localhost] ct-woodpecker: *(object+72) = 0xc00af54000
Feb  1 05:40:38 [localhost] ct-woodpecker: *(object+80) = 0x2
Feb  1 05:40:38 [localhost] ct-woodpecker: *(object+88) = 0x7ea
Feb  1 05:40:38 [localhost] ct-woodpecker: fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime stack:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.throw(0xb7b487, 0x3e)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/panic.go:608 +0x72 fp=0x7f5cda1c8d20 sp=0x7f5cda1c8cf0 pc=0x4366d2
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.findObject(0xc002b3a600, 0xc0080483c0, 0x20, 0x7f5cd8d8bd30, 0xc00003b270, 0x5b)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mbitmap.go:399 +0x3b6 fp=0x7f5cda1c8d70 sp=0x7f5cda1c8d20 pc=0x41ecd6
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.scanobject(0xc0080483c0, 0xc00003b270)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mgcmark.go:1150 +0x233 fp=0x7f5cda1c8e00 sp=0x7f5cda1c8d70 pc=0x429bc3
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gcDrain(0xc00003b270, 0x15)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mgcmark.go:913 +0x23e fp=0x7f5cda1c8e58 sp=0x7f5cda1c8e00 pc=0x42941e
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gcBgMarkWorker.func2()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mgc.go:1860 +0x1a4 fp=0x7f5cda1c8e98 sp=0x7f5cda1c8e58 pc=0x45f324
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.systemstack(0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:351 +0x66 fp=0x7f5cda1c8ea0 sp=0x7f5cda1c8e98 pc=0x461856
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.mstart()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:1229 fp=0x7f5cda1c8ea8 sp=0x7f5cda1c8ea0 pc=0x43afc0
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 134 [GC worker (idle)]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.systemstack_switch()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc000292f60 sp=0xc000292f58 pc=0x4617e0
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gcBgMarkWorker(0xc00003a000)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mgc.go:1826 +0x1b8 fp=0xc000292fd8 sp=0xc000292f60 pc=0x426078
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000292fe0 sp=0xc000292fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by runtime.gcBgMarkStartWorkers
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mgc.go:1720 +0x77
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 1 [chan receive, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bb8, 0xc000033bb8, 0xc00003170d, 0x3)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00017cd10 sp=0xc00017ccf0 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goparkunlock(0xc000033bb8, 0x170d, 0x3)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:308 +0x53 fp=0xc00017cd40 sp=0xc00017cd10 pc=0x4384e3
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.chanrecv(0xc000033b60, 0xc000271e58, 0x4f4201, 0xc00029e160)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/chan.go:520 +0x2be fp=0xc00017cdd0 sp=0xc00017cd40 pc=0x410eee
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.chanrecv1(0xc000033b60, 0xc000271e58)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/chan.go:402 +0x2b fp=0xc00017ce00 sp=0xc00017cdd0 pc=0x410bdb
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/cmd.WaitForSignal(0xc00008f180, 0xc00017cf58)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/cmd/common.go:21 +0x15a fp=0xc00017cea8 sp=0xc00017ce00 pc=0x4f45aa
Feb  1 05:40:38 [localhost] ct-woodpecker: main.main()
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/cmd/ct-woodpecker/main.go:51 +0x1ed fp=0xc00017cf98 sp=0xc00017cea8 pc=0x9433ad
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.main()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:201 +0x207 fp=0xc00017cfe0 sp=0xc00017cf98 pc=0x438047
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00017cfe8 sp=0xc00017cfe0 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 2 [force gc (idle), 8 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bb8, 0x12e5990, 0x451410, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00004cf80 sp=0xc00004cf60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goparkunlock(0x12e5990, 0x1410, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:308 +0x53 fp=0xc00004cfb0 sp=0xc00004cf80 pc=0x4384e3
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.forcegchelper()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:251 +0xb3 fp=0xc00004cfe0 sp=0xc00004cfb0 pc=0x4382b3
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00004cfe8 sp=0xc00004cfe0 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by runtime.init.4
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:240 +0x35
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 3 [GC sweep wait]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bb8, 0x12e6ca0, 0x45140c, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00004d780 sp=0xc00004d760 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goparkunlock(0x12e6ca0, 0xbf140c, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:308 +0x53 fp=0xc00004d7b0 sp=0xc00004d780 pc=0x4384e3
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.bgsweep(0xc00006c000)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mgcsweep.go:71 +0x102 fp=0xc00004d7d8 sp=0xc00004d7b0 pc=0x42ad82
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00004d7e0 sp=0xc00004d7d8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by runtime.gcenable
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mgc.go:216 +0x58
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 4 [finalizer wait, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bb8, 0x1306320, 0x140f, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00004df28 sp=0xc00004df08 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goparkunlock(0x1306320, 0x140f, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:308 +0x53 fp=0xc00004df58 sp=0xc00004df28 pc=0x4384e3
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.runfinq()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mfinal.go:175 +0x99 fp=0xc00004dfe0 sp=0xc00004df58 pc=0x4226c9
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00004dfe8 sp=0xc00004dfe0 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by runtime.createfing
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mfinal.go:156 +0x61
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 5 [syscall, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.notetsleepg(0x1306820, 0xffffffffffffffff, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/lock_futex.go:227 +0x37 fp=0xc00004e788 sp=0xc00004e758 pc=0x415bc7
Feb  1 05:40:38 [localhost] ct-woodpecker: os/signal.signal_recv(0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/sigqueue.go:139 +0x9c fp=0xc00004e7b0 sp=0xc00004e788 pc=0x44c53c
Feb  1 05:40:38 [localhost] ct-woodpecker: os/signal.loop()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/os/signal/signal_unix.go:23 +0x22 fp=0xc00004e7e0 sp=0xc00004e7b0 pc=0x4f41b2
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00004e7e8 sp=0xc00004e7e0 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by os/signal.init.0
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/os/signal/signal_unix.go:29 +0x41
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 8 [chan receive]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bb8, 0xc000033c78, 0xc00003170d, 0x3)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00004c6a0 sp=0xc00004c680 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goparkunlock(0xc000033c78, 0x170d, 0x3)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:308 +0x53 fp=0xc00004c6d0 sp=0xc00004c6a0 pc=0x4384e3
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.chanrecv(0xc000033c20, 0xc00004c7b0, 0x12e7f01, 0x12e7ff8)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/chan.go:520 +0x2be fp=0xc00004c760 sp=0xc00004c6d0 pc=0x410eee
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.chanrecv2(0xc000033c20, 0xc00004c7b0, 0x7f7101)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/chan.go:407 +0x2b fp=0xc00004c790 sp=0xc00004c760 pc=0x410c1b
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x12e7fe0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/golang/glog/glog.go:882 +0x8b fp=0xc00004c7d8 sp=0xc00004c790 pc=0x9183cb
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00004c7e0 sp=0xc00004c7d8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/vendor/github.com/golang/glog.init.0
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/golang/glog/glog.go:410 +0x203
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 23 [select, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00004edd8 sp=0xc00004edb8 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc00004ef68, 0xc00004ef60, 0x2, 0x410b10, 0xc0000332b8)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc00004ef38 sp=0xc00004edd8 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc000104180, 0xc09020, 0xc000031080)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8 fp=0xc00004efc8 sp=0xc00004ef38 pc=0x6f4c68
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00004efd0 sp=0xc00004efc8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by database/sql.OpenDB
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 24 [select, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00004f5c0 sp=0xc00004f5a0 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc00004f768, 0xc00004f748, 0x2, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc00004f720 sp=0xc00004f5c0 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc000104180, 0xc09020, 0xc000031080)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb fp=0xc00004f7c8 sp=0xc00004f720 pc=0x6f4d9b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00004f7d0 sp=0xc00004f7c8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by database/sql.OpenDB
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 25 [select, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00004fdd8 sp=0xc00004fdb8 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc00004ff68, 0xc00004ff60, 0x2, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc00004ff38 sp=0xc00004fdd8 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc000104300, 0xc09020, 0xc0000313c0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8 fp=0xc00004ffc8 sp=0xc00004ff38 pc=0x6f4c68
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00004ffd0 sp=0xc00004ffc8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by database/sql.OpenDB
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 26 [select, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0000485c0 sp=0xc0000485a0 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000048768, 0xc000048748, 0x2, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000048720 sp=0xc0000485c0 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc000104300, 0xc09020, 0xc0000313c0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb fp=0xc0000487c8 sp=0xc000048720 pc=0x6f4d9b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000487d0 sp=0xc0000487c8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by database/sql.OpenDB
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 27 [select, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000048dd8 sp=0xc000048db8 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000048f68, 0xc000048f60, 0x2, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000048f38 sp=0xc000048dd8 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc000104480, 0xc09020, 0xc0000316c0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8 fp=0xc000048fc8 sp=0xc000048f38 pc=0x6f4c68
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000048fd0 sp=0xc000048fc8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by database/sql.OpenDB
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 28 [select, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0000495c0 sp=0xc0000495a0 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000049768, 0xc000049748, 0x2, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000049720 sp=0xc0000495c0 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc000104480, 0xc09020, 0xc0000316c0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb fp=0xc0000497c8 sp=0xc000049720 pc=0x6f4d9b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000497d0 sp=0xc0000497c8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by database/sql.OpenDB
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 29 [select, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000049dd8 sp=0xc000049db8 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000049f68, 0xc000049f60, 0x2, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000049f38 sp=0xc000049dd8 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc000104600, 0xc09020, 0xc000031a00)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8 fp=0xc000049fc8 sp=0xc000049f38 pc=0x6f4c68
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000049fd0 sp=0xc000049fc8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by database/sql.OpenDB
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 30 [select, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00004a5c0 sp=0xc00004a5a0 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc00004a768, 0xc00004a748, 0x2, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc00004a720 sp=0xc00004a5c0 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc000104600, 0xc09020, 0xc000031a00)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb fp=0xc00004a7c8 sp=0xc00004a720 pc=0x6f4d9b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00004a7d0 sp=0xc00004a7c8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by database/sql.OpenDB
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 31 [select, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00004add8 sp=0xc00004adb8 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc00004af68, 0xc00004af60, 0x2, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc00004af38 sp=0xc00004add8 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc000104780, 0xc09020, 0xc000031d00)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8 fp=0xc00004afc8 sp=0xc00004af38 pc=0x6f4c68
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] systemd: ct-woodpecker.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00004afd0 sp=0xc00004afc8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by database/sql.OpenDB
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 32 [select, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00004b5c0 sp=0xc00004b5a0 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc00004b768, 0xc00004b748, 0x2, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc00004b720 sp=0xc00004b5c0 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc000104780, 0xc09020, 0xc000031d00)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb fp=0xc00004b7c8 sp=0xc00004b720 pc=0x6f4d9b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00004b7d0 sp=0xc00004b7c8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by database/sql.OpenDB
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 33 [select, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00004bdd8 sp=0xc00004bdb8 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc00004bf68, 0xc00004bf60, 0x2, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc00004bf38 sp=0xc00004bdd8 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc000104900, 0xc09020, 0xc000290040)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8 fp=0xc00004bfc8 sp=0xc00004bf38 pc=0x6f4c68
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00004bfd0 sp=0xc00004bfc8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by database/sql.OpenDB
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 34 [select, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0002965c0 sp=0xc0002965a0 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000296768, 0xc000296748, 0x2, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000296720 sp=0xc0002965c0 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc000104900, 0xc09020, 0xc000290040)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb fp=0xc0002967c8 sp=0xc000296720 pc=0x6f4d9b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] systemd: Unit ct-woodpecker.service entered failed state.
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0002967d0 sp=0xc0002967c8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by database/sql.OpenDB
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 35 [select, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000296dd8 sp=0xc000296db8 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000296f68, 0xc000296f60, 0x2, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000296f38 sp=0xc000296dd8 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc000104a80, 0xc09020, 0xc000290340)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8 fp=0xc000296fc8 sp=0xc000296f38 pc=0x6f4c68
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000296fd0 sp=0xc000296fc8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by database/sql.OpenDB
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 36 [select, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0002975c0 sp=0xc0002975a0 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000297768, 0xc000297748, 0x2, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000297720 sp=0xc0002975c0 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc000104a80, 0xc09020, 0xc000290340)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb fp=0xc0002977c8 sp=0xc000297720 pc=0x6f4d9b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0002977d0 sp=0xc0002977c8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by database/sql.OpenDB
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 37 [select, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000297dd8 sp=0xc000297db8 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000297f68, 0xc000297f60, 0x2, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000297f38 sp=0xc000297dd8 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc000104c00, 0xc09020, 0xc000290640)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8 fp=0xc000297fc8 sp=0xc000297f38 pc=0x6f4c68
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] systemd: ct-woodpecker.service failed.
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000297fd0 sp=0xc000297fc8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by database/sql.OpenDB
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 38 [select, 590 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0002985c0 sp=0xc0002985a0 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000298768, 0xc000298748, 0x2, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000298720 sp=0xc0002985c0 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc000104c00, 0xc09020, 0xc000290640)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb fp=0xc0002987c8 sp=0xc000298720 pc=0x6f4d9b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0002987d0 sp=0xc0002987c8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by database/sql.OpenDB
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 41 [IO wait]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86b88, 0x7f5ce00b8f28, 0x1b02, 0x5)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00005fb80 sp=0xc00005fb60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.netpollblock(0x7f5ce00b8f00, 0x72, 0xc0001f1500)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/netpoll.go:366 +0x99 fp=0xc00005fbb8 sp=0xc00005fb80 pc=0x433769
Feb  1 05:40:38 [localhost] ct-woodpecker: internal/poll.runtime_pollWait(0x7f5ce00b8f00, 0x72, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/netpoll.go:173 +0x66 fp=0xc00005fbe8 sp=0xc00005fbb8 pc=0x432e16
Feb  1 05:40:38 [localhost] ct-woodpecker: internal/poll.(*pollDesc).wait(0xc000103618, 0x72, 0xc000030100, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a fp=0xc00005fc18 sp=0xc00005fbe8 pc=0x49bdca
Feb  1 05:40:38 [localhost] ct-woodpecker: internal/poll.(*pollDesc).waitRead(0xc000103618, 0xffffffffffffff00, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d fp=0xc00005fc50 sp=0xc00005fc18 pc=0x49bedd
Feb  1 05:40:38 [localhost] ct-woodpecker: internal/poll.(*FD).Accept(0xc000103600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1a0 fp=0xc00005fcb8 sp=0xc00005fc50 pc=0x49e1e0
Feb  1 05:40:38 [localhost] ct-woodpecker: net.(*netFD).accept(0xc000103600, 0x416b9f, 0xc0004ba820, 0xa0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/net/fd_unix.go:238 +0x42 fp=0xc00005fd88 sp=0xc00005fcb8 pc=0x58f4a2
Feb  1 05:40:38 [localhost] ct-woodpecker: net.(*TCPListener).accept(0xc00000e498, 0xc00005fe00, 0x9d0cab28, 0x7061352bd0ab556a)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/net/tcpsock_posix.go:139 +0x2e fp=0xc00005fdb8 sp=0xc00005fd88 pc=0x5abe2e
Feb  1 05:40:38 [localhost] ct-woodpecker: net.(*TCPListener).AcceptTCP(0xc00000e498, 0xc00005fe28, 0x4933b6, 0x5c53dbc2)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/net/tcpsock.go:247 +0x47 fp=0xc00005fdf8 sp=0xc00005fdb8 pc=0x5aa867
Feb  1 05:40:38 [localhost] ct-woodpecker: net/http.tcpKeepAliveListener.Accept(0xc00000e498, 0xc00005fe78, 0x18, 0xc0001f1500, 0x6bd135)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/net/http/server.go:3232 +0x2f fp=0xc00005fe30 sp=0xc00005fdf8 pc=0x6bec3f
Feb  1 05:40:38 [localhost] ct-woodpecker: net/http.(*onceCloseListener).Accept(0xc0001e7350, 0xb865a8, 0xc0004ba820, 0xc090e0, 0xc0001e7410)
Feb  1 05:40:38 [localhost] ct-woodpecker: <autogenerated>:1 +0x3c fp=0xc00005fe68 sp=0xc00005fe30 pc=0x6df7dc
Feb  1 05:40:38 [localhost] ct-woodpecker: net/http.(*Server).Serve(0xc000157520, 0xc08ba0, 0xc00000e498, 0x0, 0x0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/net/http/server.go:2826 +0x22f fp=0xc00005ff28 sp=0xc00005fe68 pc=0x6bd06f
Feb  1 05:40:38 [localhost] ct-woodpecker: net/http.(*Server).ListenAndServe(0xc000157520, 0x0, 0xc000298f90)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/net/http/server.go:2764 +0xb6 fp=0xc00005ff78 sp=0xc00005ff28 pc=0x6bcd26
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/woodpecker.(*Woodpecker).Run.func1(0xc0001e7290)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/woodpecker/woodpecker.go:396 +0x33 fp=0xc00005ffd8 sp=0xc00005ff78 pc=0x9428e3
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00005ffe0 sp=0xc00005ffd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/woodpecker.(*Woodpecker).Run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/woodpecker/woodpecker.go:395 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 42 [select, 1 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0003a3d80 sp=0xc0003a3d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc0003a3f78, 0xc0003a3f58, 0x2, 0x15180, 0xb5da6d)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc0003a3ee0 sp=0xc0003a3d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc000141040)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192 fp=0xc0003a3fd8 sp=0xc0003a3ee0 pc=0x92b1d2
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0003a3fe0 sp=0xc0003a3fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 43 [select, 10 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000319d80 sp=0xc000319d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000319f78, 0xc000319f58, 0x2, 0x15180, 0xb5f270)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000319ee0 sp=0xc000319d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc000104240)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184 fp=0xc000319fd8 sp=0xc000319ee0 pc=0x92ad34
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000319fe0 sp=0xc000319fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 44 [select]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0003a1d80 sp=0xc0003a1d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc0003a1f78, 0xc0003a1f58, 0x2, 0x1, 0xb60f01)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc0003a1ee0 sp=0xc0003a1d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc000141220)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0 fp=0xc0003a1fd8 sp=0xc0003a1ee0 pc=0x92af60
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0003a1fe0 sp=0xc0003a1fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 45 [select, 1 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000060d80 sp=0xc000060d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000060f78, 0xc000060f58, 0x2, 0x15180, 0xb5da6d)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000060ee0 sp=0xc000060d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc000141360)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192 fp=0xc000060fd8 sp=0xc000060ee0 pc=0x92b1d2
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000060fe0 sp=0xc000060fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 46 [select]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0006d7d80 sp=0xc0006d7d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc0006d7f78, 0xc0006d7f58, 0x2, 0x15180, 0xb5f270)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc0006d7ee0 sp=0xc0006d7d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc0001043c0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184 fp=0xc0006d7fd8 sp=0xc0006d7ee0 pc=0x92ad34
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0006d7fe0 sp=0xc0006d7fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 47 [select]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0006d5d80 sp=0xc0006d5d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc0006d5f78, 0xc0006d5f58, 0x2, 0x15180, 0xb60fd7)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc0006d5ee0 sp=0xc0006d5d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc000141540)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0 fp=0xc0006d5fd8 sp=0xc0006d5ee0 pc=0x92af60
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0006d5fe0 sp=0xc0006d5fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 48 [select, 2 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000061d80 sp=0xc000061d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000061f78, 0xc000061f58, 0x2, 0x15180, 0xb5da6d)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000061ee0 sp=0xc000061d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc000141680)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192 fp=0xc000061fd8 sp=0xc000061ee0 pc=0x92b1d2
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000061fe0 sp=0xc000061fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 49 [select, 10 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00039fd80 sp=0xc00039fd60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc00039ff78, 0xc00039ff58, 0x2, 0x15180, 0xb5f270)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc00039fee0 sp=0xc00039fd80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc000104540)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184 fp=0xc00039ffd8 sp=0xc00039fee0 pc=0x92ad34
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00039ffe0 sp=0xc00039ffd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 50 [select]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00039dd80 sp=0xc00039dd60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc00039df78, 0xc00039df58, 0x2, 0x1, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc00039dee0 sp=0xc00039dd80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc000141860)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0 fp=0xc00039dfd8 sp=0xc00039dee0 pc=0x92af60
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00039dfe0 sp=0xc00039dfd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 51 [select, 2 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00005ad80 sp=0xc00005ad60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc00005af78, 0xc00005af58, 0x2, 0x15180, 0xb5da6d)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc00005aee0 sp=0xc00005ad80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc0001419a0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192 fp=0xc00005afd8 sp=0xc00005aee0 pc=0x92b1d2
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00005afe0 sp=0xc00005afd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 52 [select]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0003b7d80 sp=0xc0003b7d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc0003b7f78, 0xc0003b7f58, 0x2, 0x15180, 0xb5f270)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc0003b7ee0 sp=0xc0003b7d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc0001046c0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184 fp=0xc0003b7fd8 sp=0xc0003b7ee0 pc=0x92ad34
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0003b7fe0 sp=0xc0003b7fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 53 [select]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000312d80 sp=0xc000312d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000312f78, 0xc000312f58, 0x2, 0x1, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000312ee0 sp=0xc000312d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc000141b80)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0 fp=0xc000312fd8 sp=0xc000312ee0 pc=0x92af60
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000312fe0 sp=0xc000312fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 54 [select, 2 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00005bd80 sp=0xc00005bd60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc00005bf78, 0xc00005bf58, 0x2, 0x15180, 0xb5da6d)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc00005bee0 sp=0xc00005bd80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc000141cc0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192 fp=0xc00005bfd8 sp=0xc00005bee0 pc=0x92b1d2
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00005bfe0 sp=0xc00005bfd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 55 [select, 10 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000318d80 sp=0xc000318d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000318f78, 0xc000318f58, 0x2, 0x15180, 0xb5f270)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000318ee0 sp=0xc000318d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc000104840)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184 fp=0xc000318fd8 sp=0xc000318ee0 pc=0x92ad34
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000318fe0 sp=0xc000318fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 56 [select]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000314d80 sp=0xc000314d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000314f78, 0xc000314f58, 0x2, 0x1, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000314ee0 sp=0xc000314d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc000141ea0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0 fp=0xc000314fd8 sp=0xc000314ee0 pc=0x92af60
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000314fe0 sp=0xc000314fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 57 [select, 1 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00005cd80 sp=0xc00005cd60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc00005cf78, 0xc00005cf58, 0x2, 0x15180, 0xb5da6d)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc00005cee0 sp=0xc00005cd80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc00029a000)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192 fp=0xc00005cfd8 sp=0xc00005cee0 pc=0x92b1d2
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00005cfe0 sp=0xc00005cfd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 58 [select]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0003b3d80 sp=0xc0003b3d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc0003b3f78, 0xc0003b3f58, 0x2, 0x15180, 0xb5f270)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc0003b3ee0 sp=0xc0003b3d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc0001049c0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184 fp=0xc0003b3fd8 sp=0xc0003b3ee0 pc=0x92ad34
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0003b3fe0 sp=0xc0003b3fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 59 [select]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000317d80 sp=0xc000317d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000317f78, 0xc000317f58, 0x2, 0x1, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000317ee0 sp=0xc000317d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc00029a1e0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0 fp=0xc000317fd8 sp=0xc000317ee0 pc=0x92af60
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000317fe0 sp=0xc000317fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 60 [select, 1 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00005dd80 sp=0xc00005dd60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc00005df78, 0xc00005df58, 0x2, 0x15180, 0xb5da6d)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc00005dee0 sp=0xc00005dd80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc00029a320)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192 fp=0xc00005dfd8 sp=0xc00005dee0 pc=0x92b1d2
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00005dfe0 sp=0xc00005dfd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 61 [select, 10 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0003a2d80 sp=0xc0003a2d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc0003a2f78, 0xc0003a2f58, 0x2, 0x15180, 0xb5f270)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc0003a2ee0 sp=0xc0003a2d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc000104b40)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184 fp=0xc0003a2fd8 sp=0xc0003a2ee0 pc=0x92ad34
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0003a2fe0 sp=0xc0003a2fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 62 [select]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000315d80 sp=0xc000315d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000315f78, 0xc000315f58, 0x2, 0x1, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000315ee0 sp=0xc000315d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc00029a500)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0 fp=0xc000315fd8 sp=0xc000315ee0 pc=0x92af60
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000315fe0 sp=0xc000315fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 63 [select, 1 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0003a0d80 sp=0xc0003a0d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc0003a0f78, 0xc0003a0f58, 0x2, 0x15180, 0xb5da6d)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc0003a0ee0 sp=0xc0003a0d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc00029a640)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192 fp=0xc0003a0fd8 sp=0xc0003a0ee0 pc=0x92b1d2
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0003a0fe0 sp=0xc0003a0fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 64 [select, 10 minutes]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000316d80 sp=0xc000316d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000316f78, 0xc000316f58, 0x2, 0x15180, 0xb5f270)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000316ee0 sp=0xc000316d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc000104cc0)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184 fp=0xc000316fd8 sp=0xc000316ee0 pc=0x92ad34
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000316fe0 sp=0xc000316fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 65 [select]:
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.gopark(0xb86bf8, 0x0, 0x1809, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000313d80 sp=0xc000313d60 pc=0x43843b
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.selectgo(0xc000313f78, 0xc000313f58, 0x2, 0x1, 0x1)
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/select.go:313 +0xcc6 fp=0xc000313ee0 sp=0xc000313d80 pc=0x447a16
Feb  1 05:40:38 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc00029a820)
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0 fp=0xc000313fd8 sp=0xc000313ee0 pc=0x92af60
Feb  1 05:40:38 [localhost] ct-woodpecker: runtime.goexit()
Feb  1 05:40:38 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000313fe0 sp=0xc000313fd8 pc=0x4638c1
Feb  1 05:40:38 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Feb  1 05:40:38 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Feb  1 05:40:38 [localhost] ct-woodpecker: goroutine 66 [select, 590 min

Inclusion checker: Resubmit non-ct-woodpecker certs from get-entires

It would be nice to resubmit certificates from a monitored log beyond ones issued & previously submitted by a ct-woodpecker instance. E.g. pick random entries from the log, resubmit them, track for inclusion. This is helpful for monitoring logs that aren't configured with the ct-woodpecker issuer in the log's accepted roots.

Avoid coordinated omission

Right now we fetch STH, then sleep for an interval, then fetch another STH. When fetching is fast, this will fetch an STH every interval. However, if fetching an STH is slow for any reason, we'll fetch an STH every interval + fetch time. When a log is having problems, this can produce a skewed view of how many failed requests the outside world is seeing.

We should instead kick off a fresh STH fetch every interval in a new goroutine, rain or shine. We should do the same with certificate submissions once that's landed.

## Pull Request Test Coverage Report for [Build 1134](https://coveralls.io/builds/29749377)

Pull Request Test Coverage Report for Build 1134

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 69.243%

Totals
Change from base Build 1012:	0.0%
Covered Lines:	860
Relevant Lines:	1242

💛 - Coveralls

Originally posted by @coveralls in #111 (comment)

Store log name in database in addition to LogID

When looking at the LogIndexes table, I see the following information.

+----------------------+-----------+
| LogID                | LogIndex  |
+----------------------+-----------+
| -1231231231231231231 |    123 |
+----------------------+-----------+

It's easy to determine what CT log is related to which ct-woodpecker ID when there are only a few monitored logs. When there's many logs, it becomes a burden trying to figure out what LogID corresponds to what CT log.

Investigate/fix InclusionChecker Out-of-memory crash

InclusionChecker OOM crash:

Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: fatal error: runtime: out of memory
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime stack:
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.throw(0xafc1a7, 0x16)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/panic.go:616 +0x81
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.sysMap(0xc4dbe30000, 0x400000, 0x0, 0x10be258)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/mem_linux.go:216 +0x20a
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.(*mheap).sysAlloc(0x10a4580, 0x400000, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/malloc.go:470 +0xd4
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.(*mheap).grow(0x10a4580, 0x200, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/mheap.go:907 +0x60
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.(*mheap).allocSpanLocked(0x10a4580, 0x200, 0x10be268, 0x109e5f0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/mheap.go:820 +0x301
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.(*mheap).alloc_m(0x10a4580, 0x200, 0x3f8ea0101, 0x1)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/mheap.go:686 +0x118
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.(*mheap).alloc.func1()
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/mheap.go:753 +0x4d
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.(*mheap).alloc(0x10a4580, 0x200, 0xc420010101, 0xb1c198)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/mheap.go:752 +0x8a
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.largeAlloc(0x3ffe00, 0xc420460101, 0x1)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/malloc.go:826 +0x94
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.mallocgc.func1()
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/malloc.go:721 +0x46
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.systemstack(0x7fde00000000)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/asm_amd64.s:409 +0x79
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.mstart()
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/proc.go:1175
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: goroutine 30 [running]:
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.systemstack_switch()
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/asm_amd64.s:363 fp=0xc420227418 sp=0xc420227410 pc=0x45d180
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.mallocgc(0x3ffe00, 0x9f17a0, 0x1, 0xc4202274f0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/malloc.go:720 +0x8a2 fp=0xc4202274b8 sp=0xc420227418 pc=0x419c02
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.makeslice(0x9f17a0, 0x3ffe00, 0x3ffe00, 0x654aec, 0xc4db954c40, 0xc4dbcf1e00)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/slice.go:61 +0x77 fp=0xc4202274e8 sp=0xc4202274b8 pc=0x449267
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: bytes.makeSlice(0x3ffe00, 0x0, 0x0, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/bytes/buffer.go:230 +0x6d fp=0xc420227528 sp=0xc4202274e8 pc=0x4f4b5d
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: bytes.(*Buffer).grow(0xc4db7790a0, 0x200, 0x100000)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/bytes/buffer.go:144 +0x151 fp=0xc420227578 sp=0xc420227528 pc=0x4f4511
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: bytes.(*Buffer).ReadFrom(0xc4db7790a0, 0x7fdea6e1bb90, 0xc4db9eeb80, 0xc4db060780, 0xc4db060800, 0xc4d851cc00)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/bytes/buffer.go:204 +0x48 fp=0xc4202275e8 sp=0xc420227578 pc=0x4f4978
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: io/ioutil.readAll(0x7fdea6e1bb90, 0xc4db9eeb80, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/io/ioutil/ioutil.go:36 +0xb5 fp=0xc420227630 sp=0xc4202275e8 pc=0x55ba75
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: io/ioutil.ReadAll(0x7fdea6e1bb90, 0xc4db9eeb80, 0xc4db9eeb80, 0x7fdea6e1bb90, 0xc4db9eeb80, 0x0, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/io/ioutil/ioutil.go:45 +0x3e fp=0xc420227680 sp=0xc420227630 pc=0x55bb3e
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: github.com/google/certificate-transparency-go/jsonclient.(*JSONClient).GetAndParse(0xc420062a80, 0xb68140, 0xc420026028, 0xaf91e4, 0x12, 0xc420227a30, 0x9cf5c0, 0xc4db9eea20, 0x0, 0xc4d891c000, ...)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/gopkg/src/github.com/google/certificate-transparency-go/jsonclient/client.go:173 +0x4a7 fp=0xc420227998 sp=0xc420227680 pc=0x7c0117
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: github.com/google/certificate-transparency-go/client.(*LogClient).GetRawEntries(0xc420062a80, 0xb68140, 0xc420026028, 0x9e85, 0xa079, 0x0, 0x0, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/gopkg/src/github.com/google/certificate-transparency-go/client/getentries.go:44 +0x2cd fp=0xc420227bb0 sp=0xc420227998 pc=0x7c457d
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: github.com/google/certificate-transparency-go/client.(*LogClient).GetEntries(0xc420062a80, 0xb68140, 0xc420026028, 0x9e85, 0xa079, 0x1f5, 0x1f5, 0x1f5, 0x0, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/gopkg/src/github.com/google/certificate-transparency-go/client/getentries.go:61 +0x74 fp=0xc420227c98 sp=0xc420227bb0 pc=0x7c47d4
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).getEntries(0xc42011e480, 0x9e85, 0x11fe2081, 0xc4204f8300, 0x0, 0x0, 0x0, 0x1, 0x1)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/gopkg/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:140 +0x164 fp=0xc420227d38 sp=0xc420227c98 pc=0x8c86b4
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).checkInclusion(0xc42011e480, 0x1, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/gopkg/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:105 +0x555 fp=0xc420227ef0 sp=0xc420227d38 pc=0x8c8085
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc42011e480)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/gopkg/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:68 +0x11f fp=0xc420227fd8 sp=0xc420227ef0 pc=0x8ccc8f
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.goexit()
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420227fe0 sp=0xc420227fd8 pc=0x45fcf1
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/gopkg/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:61 +0x3f
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: goroutine 1 [chan receive]:
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: github.com/letsencrypt/ct-woodpecker/cmd.WaitForSignal(0xc420020f00, 0xc42004ef48)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/gopkg/src/github.com/letsencrypt/ct-woodpecker/cmd/common.go:21 +0x15a
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: main.main()
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/gopkg/src/github.com/letsencrypt/ct-woodpecker/cmd/ct-woodpecker/main.go:47 +0x1d3
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: goroutine 5 [syscall]:
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: os/signal.signal_recv(0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/sigqueue.go:139 +0xa6
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: os/signal.loop()
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/os/signal/signal_unix.go:22 +0x22
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: created by os/signal.init.0
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/os/signal/signal_unix.go:28 +0x41
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: goroutine 8 [chan receive]:
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: github.com/golang/glog.(*loggingT).flushDaemon(0x109e520)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/gopkg/src/github.com/golang/glog/glog.go:882 +0x8b
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: created by github.com/golang/glog.init.0
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/gopkg/src/github.com/golang/glog/glog.go:410 +0x203
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: goroutine 23 [select]:
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: database/sql.(*DB).connectionOpener(0xc420087d60, 0xb68100, 0xc420062ac0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/database/sql/sql.go:935 +0x119
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: created by database/sql.OpenDB
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/database/sql/sql.go:634 +0x178
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: goroutine 24 [select]:
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: database/sql.(*DB).connectionResetter(0xc420087d60, 0xb68100, 0xc420062ac0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/database/sql/sql.go:948 +0x12a
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: created by database/sql.OpenDB
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/database/sql/sql.go:635 +0x1ae
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: goroutine 27 [IO wait]:
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: internal/poll.runtime_pollWait(0x7fdea6dfaf00, 0x72, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/netpoll.go:173 +0x57
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: internal/poll.(*pollDesc).wait(0xc42011e518, 0x72, 0xc420062000, 0x0, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: internal/poll.(*pollDesc).waitRead(0xc42011e518, 0xffffffffffffff00, 0x0, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: internal/poll.(*FD).Accept(0xc42011e500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/internal/poll/fd_unix.go:372 +0x1a8
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: net.(*netFD).accept(0xc42011e500, 0xa14720, 0xc42004de10, 0x40a848)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/fd_unix.go:238 +0x42
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: net.(*TCPListener).accept(0xc42000e198, 0xc42004de40, 0x4097a7, 0xc442663660)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/tcpsock_posix.go:136 +0x2e
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: net.(*TCPListener).AcceptTCP(0xc42000e198, 0xc42004de88, 0xc42004de90, 0x18)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/tcpsock.go:246 +0x49
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: net/http.tcpKeepAliveListener.Accept(0xc42000e198, 0xb1bbe8, 0xc4426635e0, 0xb681c0, 0xc420163c80)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/http/server.go:3216 +0x2f
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: net/http.(*Server).Serve(0xc420190d00, 0xb67f40, 0xc42000e198, 0x0, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/http/server.go:2770 +0x1a5
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: net/http.(*Server).ListenAndServe(0xc420190d00, 0xc420066d18, 0xc4200367b8)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/http/server.go:2711 +0xa9
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: github.com/letsencrypt/ct-woodpecker/woodpecker.(*Woodpecker).Run.func1(0xc420062c00)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/gopkg/src/github.com/letsencrypt/ct-woodpecker/woodpecker/woodpecker.go:337 +0x33
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: created by github.com/letsencrypt/ct-woodpecker/woodpecker.(*Woodpecker).Run
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/gopkg/src/github.com/letsencrypt/ct-woodpecker/woodpecker/woodpecker.go:336 +0x3f
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: goroutine 28 [select]:
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc42011e400)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/gopkg/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:172 +0x185
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/gopkg/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:168 +0x3f
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: goroutine 29 [select]:
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc4201d6000)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/gopkg/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:133 +0x174
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/gopkg/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:128 +0x3f
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: goroutine 31 [select, locked to thread]:
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.gopark(0xb1c1d0, 0x0, 0xaf2136, 0x6, 0x18, 0x1)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/proc.go:291 +0x11a
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.selectgo(0xc420038750, 0xc420070840)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/select.go:392 +0xe50
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.ensureSigM.func1()
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/signal_unix.go:549 +0x1f4
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: runtime.goexit()
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/asm_amd64.s:2361 +0x1
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: goroutine 76 [IO wait]:
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: internal/poll.runtime_pollWait(0x7fdea6dfad60, 0x72, 0xc4428b9868)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/runtime/netpoll.go:173 +0x57
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: internal/poll.(*pollDesc).wait(0xc4388e7f98, 0x72, 0xffffffffffffff00, 0xb61940, 0x1061378)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: internal/poll.(*pollDesc).waitRead(0xc4388e7f98, 0xc440621000, 0x1000, 0x1000)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: internal/poll.(*FD).Read(0xc4388e7f80, 0xc440621000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/internal/poll/fd_unix.go:157 +0x17d
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: net.(*netFD).Read(0xc4388e7f80, 0xc440621000, 0x1000, 0x1000, 0xc4428b9978, 0x68c2ea, 0xc4428748a8)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/fd_unix.go:202 +0x4f
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: net.(*conn).Read(0xc42000e1c0, 0xc440621000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/net.go:176 +0x6a
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: net/http.(*connReader).Read(0xc4428748a0, 0xc440621000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/http/server.go:764 +0xf8
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: bufio.(*Reader).fill(0xc4427d2540)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/bufio/bufio.go:100 +0x11e
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: bufio.(*Reader).ReadSlice(0xc4427d2540, 0xc4201d930a, 0xc4428b9ad8, 0x4191c6, 0x7fdea42e0990, 0x4, 0xc4428b9ac0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/bufio/bufio.go:341 +0x2c
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: bufio.(*Reader).ReadLine(0xc4427d2540, 0xc42deb0400, 0x100, 0xf8, 0xadfa80, 0xb1b901, 0x100c4428b9b58)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/bufio/bufio.go:370 +0x34
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: net/textproto.(*Reader).readLineSlice(0xc4428748d0, 0xc48f2da080, 0xc4428b9ba8, 0x419ea8, 0x100, 0xadfa80)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/textproto/reader.go:55 +0x70
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: net/textproto.(*Reader).ReadLine(0xc4428748d0, 0xc42deb0400, 0x0, 0x0, 0x72)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/textproto/reader.go:36 +0x2b
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: net/http.readRequest(0xc4427d2540, 0x0, 0xc42deb0400, 0x0, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/http/request.go:929 +0x8c
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: net/http.(*conn).readRequest(0xc4426635e0, 0xb68100, 0xc44276d800, 0x0, 0x0, 0x0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/http/server.go:944 +0x16f
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: net/http.(*conn).serve(0xc4426635e0, 0xb68100, 0xc44276d800)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/http/server.go:1768 +0x4dc
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: created by net/http.(*Server).Serve
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/http/server.go:2795 +0x27b
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: goroutine 55 [select]:
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: net/http.(*persistConn).readLoop(0xc42018efc0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/http/transport.go:1717 +0x743
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: created by net/http.(*Transport).dialConn
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/http/transport.go:1237 +0x95a
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: goroutine 150 [select]:
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: net/http.setRequestCancel.func3(0x0, 0xc4db8ffec0, 0xc4db9de780, 0xc4db9bf968, 0xc4db97e3c0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/http/client.go:323 +0xfa
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: created by net/http.setRequestCancel
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/http/client.go:322 +0x2ac
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: goroutine 56 [select]:
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: net/http.(*persistConn).writeLoop(0xc42018efc0)
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/http/transport.go:1822 +0x14b
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]: created by net/http.(*Transport).dialConn
Jul 30 20:16:44 woodpecker ct-woodpecker[31191]:         /home/wp/go/src/net/http/transport.go:1238 +0x97f
Jul 30 20:16:44 woodpecker systemd[1]: ct-woodpecker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 30 20:16:44 woodpecker systemd[1]: ct-woodpecker.service: Unit entered failed state.
Jul 30 20:16:44 woodpecker systemd[1]: ct-woodpecker.service: Failed with result 'exit-code'.

monitor: submit precerts as well as final certs

follow up to #2

Write in-memory CT log server for integration tests

Pebble for Trillian

cttestsrv: Fix usage of deprecated Trillian protobuf fields

GolangCI's "megacheck" linter found three deprecated fields to address in the cttestsrv:

signedLogRoot.TreeSize is deprecated: TreeSize moved to LogRoot.
signedLogRoot.TimestampNanos is deprecated: TimestampNanos moved to LogRoot.
signedLogRoot.RootHash is deprecated: RootHash moved to LogRoot.

I think fixing these will require unserializing the log_root []byte from the response and referencing the fields on that structure.

Poke log API for 500s

Go through the CT RFC and try sending various malformed inputs to see what happens.

Invalid/non-linear STHs are ignored as long as they have a smaller tree size than current

This was introduced to allow us to keep operating when a log serves cached STHs that are smaller than the one we most recently saw (i.e. when they are serving stale STHs from M of N frontend nodes). This prevents us from catching a number of broken log cases.

Instead we should store a list of all STHs we've previously observed and verify the returned STHs against this list (using the existing storage layer to manage the list). This will allow us to spot bad STHs.

Set up test instance of woodpecker

Configurable testCertDomain

The testCertDomain should be configurable so that other entities besides Let's Encrypt can use their own version of ct-woodpecker. https://github.com/letsencrypt/ct-woodpecker/blob/master/pki/certs.go#L22-L26

Migrate to transparency-dev/merkle

Hi team,

I observe you are using a mix of merkle packages:

one from Trillian, and
https://github.com/transparency-dev/merkle.

We are in the process of decommissioning the merkle package from Trillian, as all our code (including Trillian itself) is now using transparency-dev/merkle. The new repository is well maintained, and we are also going to document some best practices on Merkle tree implementations.

Consider migrating your code fully to transparency-dev/merkle. Happy to help if you have questions on how to do it.

Thank you.

Keep track of SCTs from submissions and check log for inclusion

ct-woodpecker should keep a list of recently-submitted SCTs, and verify that they are eventually included in the log. Once the SCTs have been spotted in the log, they can be deleted from local storage. ct-woodpecker should export a stat indicating the size of the backlog of SCTs to check. If that backlog grows too large, there is a problem. Alternately / in addition, it could export a stat showing the age (or timestamp) of the oldest SCT that hasn't yet been seen in the log.

Investigate possible memory leak

Use vendored dependencies

We should convert this repo to use Go modules & to vendor dependencies. This will avoid upstream drift from breaking the build. (E.g. #72)

Auto-populate storage database schema

It would be nice if the storage implementation could populate the initial database schema automatically when required.

Add coverage testing to Travis

Handle submission to time sharded logs

The certSubmitter monitor doesn't support submitting to time sharded logs that gate submissions based on certificate expiry year. Presently all ct-woodpecker issued certs expire 90 days in the future:

ct-woodpecker/pki/certs.go

Line 131 in 85a085a

NotAfter: clk.Now().AddDate(0, 0, 90),

Submit new certificates in multiples

Follow up to #5

We should also have the certSubmitter submit multiple copies of the same new cert in parallel. If distinct SCTs are returned then all of them should be stored and checked for inclusion.

inclusionChecker: rare possibility of getting ahead of submissions

We have encountered the very rare occasion where the inclusionChecker is following very closely with the log/shard tree-head and happens to miss an included certificate. Re-starting ct-woodpecker with the LogIndex set back a couple tens of thousands allows ct-woodpecker to "see" the certificate it missed, and then continue on as normal.

I don't yet know how it could have missed the certificate it submitted, only that ct-woodpecker at the time was very up-to-date, following the shard's tree-head quite closely (ct-woodpecker itself may have been the only entity submitting to this shard).

Log errors to stderr

Right now we log errors to os.Stdout with a scary error prefix. We should send error output to os.Stderr instead.

Implementing this will require figuring out a good way for the integration test woodpeckerRun function to get the os.Stderr contents programmatically. Presently we use a logger backed by a buffer:

ct-woodpecker/test/integration_test.go

Lines 69 to 73 in 37199f2

 // Create a logger backed by the SafeBuffer. The log.Logger type is only safe 

 // for concurrent use when the backing buffer is. Using a raw bytes.Buffer 

 // with a shared logger will cause data races. 

 var out SafeBuffer 

 logger := log.New(&out, "integration test ", log.LstdFlags)

and call the buffer's String() method to get what we call stdout:

ct-woodpecker/test/integration_test.go

Line 90 in 37199f2

return out.String(), getWoodpeckerMetrics("http://localhost:1971"), nil

Submit duplicate certificates and precertificates

To catch conditions like this non-incorporation of SCTs for duplicate certificates: https://groups.google.com/a/chromium.org/forum/#!topic/ct-policy/E88pjOZzkIM

Cert submitter errors serializing SCTs to storage

When the certSubmitter has storage configured it tries to serialize the SCT from cert submissions to store in the storage:

ct-woodpecker/monitor/cert_submitter.go

Line 219 in 6313966

sctBytes, err := cttls.Marshal(sct)

Because the sct passed to cttls.Marshal is a pointer this produces an error:

ct-woodpecker 2018/07/22 11:07:26 !!! Error serializing SCT: tls: structure error: unsupported type: *ct.SignedCertificateTimestamp of kind ptr

Fix `cttestsrv` breakage from NewSequencer api change

test/cttestsrv/log.go:117:31: cannot use timeSource (type util.SystemTimeSource) as type clock.TimeSource in argument to "github.com/google/trillian/log".NewSequencer:
    util.SystemTimeSource does not implement clock.TimeSource (missing NewTimer method)

Similar to #69 we're fallen behind with the latest git master of trillian and so ctttestsrv will not build with a fresh go get -u of dependencies.

Write systemd unit

There should be a systemd unit for running a ct-woodpecker instance and logging to journald.

As a follow up we should consider packaging or a makefile with an install target.

Investigate unsolicited 408 time-out response.

Observed in a running ct-woodpecker instance log:

ct-woodpecker 2018/06/07 12:29:52 Certificate chain submitted to "https://birch.ct.letsencrypt.org/2018". SCT timestamp 2018-06-07 12:29:52.664 +0000 UTC
ct-woodpecker 2018/06/07 12:29:52 Sleeping for 1m0s before next certificate submission
2018/06/07 12:30:02 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n"; err=<nil>

Unclear exactly where this is coming from. It looks like its outputting on an unprefixed logger and that should be fixed too.

Log config "start" parameter named confusingly.

Am I correct in assuming that the logs config parameter start is the per log version of inclusionConfig.startIndex? What I am seeing is contrary to my assumption though and I need some clarification. Here's my logs snippet

...
  "logs": [
    {
      "uri": "https://testflume.ct.letsencrypt.org/2019"
     ,"key": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEAg3+vFOesFW51rKECekioAt9Zo50atRoOJ0qLxF7DIEHsHneXLEpgO1WMreleRy1vEbUJD7TXoH9r8qSDGvyew=="
     ,"submitPreCert": true
     ,"submitCert": true
     ,"windowStart": "2019-01-01T00:00:00Z"
     ,"windowEnd": "2020-01-01T00:00:00Z"
     ,"start": 666000000
    },
    {
      "uri": "https://testflume.ct.letsencrypt.org/2020"
     ,"key": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEjdjcoKpeBShHgHvRm3BxD5+l+eHZudv3KmD5SDcLcI01Vj5TDTmxanQKCgpvm9pfnfB6URMQV3hhU1I02jRoRw=="
     ,"submitPreCert": true
     ,"submitCert": true
     ,"windowStart": "2020-01-01T00:00:00Z"
     ,"windowEnd": "2021-01-01T00:00:00Z"
     ,"start": 1
    },
    {
      "uri": "https://testflume.ct.letsencrypt.org/2021"
     ,"key": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEdCLoJNt1QcNa7sNDp7g7oTJ+o/UIYEM6N/IZWT+dhdqtJZC+AODJ/4exdOwG04B4K6WrN1VB2ELKQIc/wU1lCw=="
     ,"submitPreCert": true
     ,"submitCert": true
     ,"windowStart": "2021-01-01T00:00:00Z"
     ,"windowEnd": "2022-01-01T00:00:00Z"
     ,"start": 1
    },
    {
      "uri": "https://testflume.ct.letsencrypt.org/2022"
     ,"key": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEjy/rXcABuf0yhrm1+XgjDnh4XPD7vfMoyJOyT+KA+c2zuXVR98yQmp/Bl5ZFdGFwJuFcVrCw7IDo0EGKs7UCww=="
     ,"submitPreCert": true
     ,"submitCert": true
     ,"windowStart": "2022-01-01T00:00:00Z"
     ,"windowEnd": "2023-01-01T00:00:00Z"
     ,"start": 1
    },
  ]
...

Here is the output I'm getting from that ct-woodpecker instance. Upon restart of the ct-woodpecker process I would expect the inclusionChecker to begin requesting entries from the start parameter per log.

2019/11/05 01:55:01 inclusionChecker https://testflume.ct.letsencrypt.org/2019 : current matches tree size. There are no new entries to check for inclusion.
2019/11/05 01:55:01 inclusionChecker https://testflume.ct.letsencrypt.org/2021 : current matches tree size. There are no new entries to check for inclusion.
2019/11/05 01:55:01 inclusionChecker https://testflume.ct.letsencrypt.org/2020 : Getting entries from 83367535 to 83367631
2019/11/05 01:55:01 inclusionChecker https://testflume.ct.letsencrypt.org/2022 : current matches tree size. There are no new entries to check for inclusion.

Add errcheck to travis

timestamp on SubmittedCerts is a bigint(20), should be datetime

In the SubmittedCerts table the timestamp field is a bigint(20). Is there a reason this couldn't be a datetime like the Seen field?

Add a "counter" component to count entries by issuer

It's handy to know who is submitting certificates at what rate. Rather than stick this into the inclusion checker, I think we should add a new component, the "counter." This component would start out from the current STH at startup, and continually run get-entries up to the current STH.

When processing entries, this component would increment a counter with three labels:

root: The Issuer CN of the topmost certificate in the entry's chain.
issuer: The Issuer CN of the end-entity certificate.
precert: True or false depending on whether the end-entity certificate is a precertificate.

	if opts.InclusionOpts != nil {
	keyHash := sha256.Sum256([]byte(opts.LogKey))
	pkBytes, err := base64.StdEncoding.DecodeString(opts.LogKey)
	if err != nil {
	return nil, err
	}
	pk, err := x509.ParsePKIXPublicKey(pkBytes)
	if err != nil {
	return nil, err
	}
	sv, err := ct.NewSignatureVerifier(pk)
	if err != nil {
	return nil, err
	}
	m.inclusionChecker = &inclusionChecker{
	logger: logger,
	client: client,
	clk: clk,
	logURI: opts.LogURI,
	interval: opts.InclusionOpts.Interval,
	db: db,
	signatureChecker: sv,
	logID: big.NewInt(0).SetBytes(keyHash[:]).Int64(),
	batchSize: opts.InclusionOpts.FetchBatchSize,
	}
	}

	var monitors []*monitor.Monitor
	for _, logConf := range c.Logs {
	opts := monitor.MonitorOptions{
	LogURI: logConf.URI,
	LogKey: logConf.Key,
	}
	if c.FetchConfig != nil {
	opts.FetchOpts = &monitor.FetcherOptions{
	Interval: fetchInterval,
	Timeout: fetchTimeout,
	}
	}
	if c.SubmitConfig != nil {
	opts.SubmitOpts = &monitor.SubmitterOptions{
	Interval: certInterval,
	Timeout: certTimeout,
	IssuerCert: issuerCert,
	IssuerKey: issuerKey,
	SubmitPreCert: logConf.SubmitPreCert,
	SubmitCert: logConf.SubmitCert,
	}
	}
	m, err := monitor.New(opts, logger, clk)
	if err != nil {
	return nil, err
	}
	monitors = append(monitors, m)
	}

	// Start the monitoring process
	woodpecker.Run()

	// Sleep for the requested amount of time
	time.Sleep(duration)

	// Collect metrics from the woodpecker instance while it is still running
	metricsData := getWoodpeckerMetrics("http://localhost:1971")

	woodpecker.Stop()

	padding := time.Millisecond * 50
	duration := submitInterval*time.Duration(iterations) + padding
	stdout, metricsData, err := woodpeckerRun(config, duration)

	// Create a logger backed by the SafeBuffer. The log.Logger type is only safe
	// for concurrent use when the backing buffer is. Using a raw bytes.Buffer
	// with a shared logger will cause data races.
	var out SafeBuffer
	logger := log.New(&out, "integration test ", log.LstdFlags)

letsencrypt / ct-woodpecker Goto Github PK

ct-woodpecker's Introduction

Getting Started

Contributing

Links

Current Features

ct-woodpecker's People

Contributors

Stargazers

Watchers

Forkers

ct-woodpecker's Issues

Pull Request Test Coverage Report for Build 1134

💛 - Coveralls

Recommend Projects

Recommend Topics

Recommend Org