What version are you running? rqlite v8.23.0 on Windows 10/11

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Use this one, which just built with my latest changes: <a href="https://ci.appveyor.co

rqlited + DNS client 100% CPU usage after network disconnecting (windows),about rqlite/rqlite

Comments (21)

otoolep commented on May 27, 2024 1

Oh, sorry, I see you are not using containers, just running on Windows. Well, a Windows build with the latest changes should be available at the link below soon.

https://ci.appveyor.com/api/projects/otoolep/rqlite/artifacts/rqlite-latest-win64.zip?branch=master

from rqlite.

otoolep commented on May 27, 2024

Thanks for the report.

I want to understand the sequence. You're saying that pulling the cable interferes somehow with the operation/behaviour of the server, such that the next time rqlited starts, it consumes 100% CPU? Is creating a cluster required to bring out this issue?

Basically this seems like something you could reproduce more simply. Is that the case?

from rqlite.

mmolder commented on May 27, 2024

The servers are connected together on a local network through a switch. I am testing some fault handling if a server were to be disconnected when running a cluster by joining and leaving it nicely and not so nicely. Pulling the ethernet cable simulates the server losing connection without having the node leave the cluster successfully. I have not seen the same behavior when running only one node.

When only two nodes are running in the cluster, disconnecting one without the issue above brings the CPU usage down on the server having the issue. Connecting it again and having it rejoin the cluster automatically brings the CPU usage back up on the previously affected server.

I have managed to reproduce this on several different servers and on the same one after flushing and resetting the DNS with the steps I described above.

from rqlite.

otoolep commented on May 27, 2024

OK, still need some clarification.

After the initial join, when the 2-node cluster is formed, everything is OK, right? In the steady state, before any disconnections, all is well, correct?

disconnecting one without the issue above brings the CPU usage

This is what is confusing me, is the issue happening after the initial join? Perhaps you could list, step-by-step, the exact sequence you are performing. E.g.:

start first node, CPU on that server is OK.
start second node, let it join to first node, CPU levels on each server is still OK
pull the cable from the second server, CPU is ....

Etc.

from rqlite.

mmolder commented on May 27, 2024

I will have to retrace the exact steps tomorrow when I have access to the servers again. In the meanwhile here are some potentially interesting output from rqlite at the time the issue started. The exceptions keep on going, this is just a handful.

[rqlited] 2024/04/03 14:14:08 rqlited starting, version 8, SQLite 3.45.1, commit unknown, branch unknown, compiler gc
[rqlited] 2024/04/03 14:14:08 go1.21.8, target architecture is amd64, operating system target is windows
[rqlited] 2024/04/03 14:14:08 launch command: rqlited -node-id 23 -http-addr 0.0.0.0:4001 -http-adv-addr x.x.x.x:4001 
-raft-addr 0.0.0.0:4002 -raft-adv-addr x.x.x.x:4002 -raft-cluster-remove-shutdown=true -raft-reap-node-timeout=15m -join y.y.y.y:4002 Path\To\db
[rqlited] 2024/04/03 14:14:08 Raft TCP mux Listener registered with byte header 1
[mux] 2024/04/03 14:14:08 mux serving on [::]:4002, advertising x.x.x.x:4002
[rqlited] 2024/04/03 14:14:08 preexisting node state detected in Path\To\db
[cluster] 2024/04/03 14:14:08 service listening on x.x.x.x:4002
[rqlited] 2024/04/03 14:14:08 cluster TCP mux Listener registered with byte header 2
[http] 2024/04/03 14:14:08 execute queue processing started with capacity 1024, batch size 128, timeout 50ms
[http] 2024/04/03 14:14:08 service listening on [::]:4001
[store] 2024/04/03 14:14:08 opening store with node ID 23, listening on x.x.x.x:4002
[store] 2024/04/03 14:14:08 ensuring data directory exists at Path\To\db
[store] 2024/04/03 14:14:08 old snapshot directory Path\To\db\snapshots does not exist, nothing to upgrade
[snapshot-store] 2024/04/03 14:14:08 store initialized using Path\To\db\rsnapshots
[snapshot-store] 2024/04/03 14:14:08 checking consistency of snapshot store at Path\To\db\rsnapshots
[snapshot-store] 2024/04/03 14:14:08 check complete
[store] 2024/04/03 14:14:08 1 preexisting snapshots present
[store] 2024/04/03 14:14:08 initiating node restore on node ID 23
[store] 2024/04/03 14:14:08 successfully opened database at Path\To\db\db.sqlite due to restore
[store] 2024/04/03 14:14:08 node restored in 157.446ms
[rqlited] 2024/04/03 14:14:08 checking that supplied join addresses don't serve HTTP(S)
[rqlited] 2024/04/03 14:14:08 successfully joined cluster at y.y.y.y:4002
[rqlited] 2024/04/03 14:14:08 node HTTP API available at http://x.x.x.x:4001
[rqlited] 2024/04/03 14:14:08 connect using the command-line tool via 'rqlite -H x.x.x.x -p 4001'
[rqlited] 2024/04/03 14:16:26 http: superfluous response.WriteHeader call from github.com/rqlite/rqlite/v8/http.(*Service).handleStatus (service.go:908)
runtime: program exceeds 10000-thread limit
fatal error: thread exhaustion

runtime stack:
runtime.throw({0x7ff62524edf4?, 0x7ff6242a1140?})
	C:/go/src/runtime/panic.go:1077 +0x65 fp=0xb4475ff7a8 sp=0xb4475ff778 pc=0x7ff62426e725
runtime.checkmcount()
	C:/go/src/runtime/proc.go:804 +0x8e fp=0xb4475ff7d0 sp=0xb4475ff7a8 pc=0x7ff6242721ce
runtime.mReserveID()
	C:/go/src/runtime/proc.go:820 +0x2f fp=0xb4475ff7f8 sp=0xb4475ff7d0 pc=0x7ff62427220f
runtime.startm(0xc000073900?, 0x0, 0x0)
	C:/go/src/runtime/proc.go:2611 +0x111 fp=0xb4475ff848 sp=0xb4475ff7f8 pc=0x7ff624275471
runtime.handoffp(0xffffffff?)
	C:/go/src/runtime/proc.go:2660 +0x2f6 fp=0xb4475ff870 sp=0xb4475ff848 pc=0x7ff624275936
runtime.retake(0x4bfc00fad744)
	C:/go/src/runtime/proc.go:5726 +0x1e8 fp=0xb4475ff8b0 sp=0xb4475ff870 pc=0x7ff62427ca88
runtime.sysmon()
	C:/go/src/runtime/proc.go:5634 +0x394 fp=0xb4475ff930 sp=0xb4475ff8b0 pc=0x7ff62427c6b4
runtime.mstart1()
	C:/go/src/runtime/proc.go:1602 +0x9d fp=0xb4475ff958 sp=0xb4475ff930 pc=0x7ff624273e1d
runtime.mstart0()
	C:/go/src/runtime/proc.go:1559 +0x6a fp=0xb4475ff980 sp=0xb4475ff958 pc=0x7ff624273d6a
runtime.mstart()
	C:/go/src/runtime/asm_amd64.s:394 +0x5 fp=0xb4475ff988 sp=0xb4475ff980 pc=0x7ff6242a1145

goroutine 1 [chan receive, 2 minutes]:
runtime.gopark(0x1b7a4232298?, 0x10?, 0x0?, 0x80?, 0xc000056dc0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc0020b7ab0 sp=0xc0020b7a90 pc=0x7ff62427144e
runtime.chanrecv(0xc0000a0420, 0x0, 0x1)
	C:/go/src/runtime/chan.go:583 +0x3d0 fp=0xc0020b7b28 sp=0xc0020b7ab0 pc=0x7ff62423adb0
runtime.chanrecv1(0x7ff625692560?, 0xc000810780?)
	C:/go/src/runtime/chan.go:442 +0x12 fp=0xc0020b7b50 sp=0xc0020b7b28 pc=0x7ff62423a9b2
main.main()
	C:/gopath/src/github.com/rqlite/rqlite/cmd/rqlited/main.go:203 +0x1777 fp=0xc0020b7f40 sp=0xc0020b7b50 pc=0x7ff624de8297
runtime.main()
	C:/go/src/runtime/proc.go:267 +0x2b2 fp=0xc0020b7fe0 sp=0xc0020b7f40 pc=0x7ff624271012
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0020b7fe8 sp=0xc0020b7fe0 pc=0x7ff6242a3221

goroutine 2 [force gc (idle), 2 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc000087fa8 sp=0xc000087f88 pc=0x7ff62427144e
runtime.goparkunlock(...)
	C:/go/src/runtime/proc.go:404
runtime.forcegchelper()
	C:/go/src/runtime/proc.go:322 +0xb8 fp=0xc000087fe0 sp=0xc000087fa8 pc=0x7ff6242712b8
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000087fe8 sp=0xc000087fe0 pc=0x7ff6242a3221
created by runtime.init.6 in goroutine 1
	C:/go/src/runtime/proc.go:310 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc000089f78 sp=0xc000089f58 pc=0x7ff62427144e
runtime.goparkunlock(...)
	C:/go/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
	C:/go/src/runtime/mgcsweep.go:321 +0xdf fp=0xc000089fc8 sp=0xc000089f78 pc=0x7ff62425991f
runtime.gcenable.func1()
	C:/go/src/runtime/mgc.go:200 +0x25 fp=0xc000089fe0 sp=0xc000089fc8 pc=0x7ff62424ea45
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000089fe8 sp=0xc000089fe0 pc=0x7ff6242a3221
created by runtime.gcenable in goroutine 1
	C:/go/src/runtime/mgc.go:200 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc000094000?, 0x7ff62567d828?, 0x0?, 0x0?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc00009bf70 sp=0xc00009bf50 pc=0x7ff62427144e
runtime.goparkunlock(...)
	C:/go/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x7ff625f25940)
	C:/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00009bfa0 sp=0xc00009bf70 pc=0x7ff624257189
runtime.bgscavenge(0x0?)
	C:/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00009bfc8 sp=0xc00009bfa0 pc=0x7ff624257739
runtime.gcenable.func2()
	C:/go/src/runtime/mgc.go:201 +0x25 fp=0xc00009bfe0 sp=0xc00009bfc8 pc=0x7ff62424e9e5
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00009bfe8 sp=0xc00009bfe0 pc=0x7ff6242a3221
created by runtime.gcenable in goroutine 1
	C:/go/src/runtime/mgc.go:201 +0xa5

goroutine 18 [finalizer wait, 2 minutes]:
runtime.gopark(0x198?, 0x7ff625236aa0?, 0x1?, 0x23?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc00008be20 sp=0xc00008be00 pc=0x7ff62427144e
runtime.runfinq()
	C:/go/src/runtime/mfinal.go:193 +0x107 fp=0xc00008bfe0 sp=0xc00008be20 pc=0x7ff62424daa7
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00008bfe8 sp=0xc00008bfe0 pc=0x7ff6242a3221
created by runtime.createfing in goroutine 1
	C:/go/src/runtime/mfinal.go:163 +0x3d

goroutine 19 [GC worker (idle), 2 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc000097f50 sp=0xc000097f30 pc=0x7ff62427144e
runtime.gcBgMarkWorker()
	C:/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000097fe0 sp=0xc000097f50 pc=0x7ff624250645
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000097fe8 sp=0xc000097fe0 pc=0x7ff6242a3221
created by runtime.gcBgMarkStartWorkers in goroutine 1
	C:/go/src/runtime/mgc.go:1219 +0x1c
	
goroutine 5 [GC worker (idle), 2 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc00009df50 sp=0xc00009df30 pc=0x7ff62427144e
runtime.gcBgMarkWorker()
	C:/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00009dfe0 sp=0xc00009df50 pc=0x7ff624250645
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00009dfe8 sp=0xc00009dfe0 pc=0x7ff6242a3221
created by runtime.gcBgMarkStartWorkers in goroutine 1
	C:/go/src/runtime/mgc.go:1219 +0x1c

goroutine 34 [GC worker (idle), 2 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc000489f50 sp=0xc000489f30 pc=0x7ff62427144e
runtime.gcBgMarkWorker()
	C:/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000489fe0 sp=0xc000489f50 pc=0x7ff624250645
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000489fe8 sp=0xc000489fe0 pc=0x7ff6242a3221
created by runtime.gcBgMarkStartWorkers in goroutine 1
	C:/go/src/runtime/mgc.go:1219 +0x1c

goroutine 6 [GC worker (idle), 2 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc000485f50 sp=0xc000485f30 pc=0x7ff62427144e
runtime.gcBgMarkWorker()
	C:/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000485fe0 sp=0xc000485f50 pc=0x7ff624250645
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000485fe8 sp=0xc000485fe0 pc=0x7ff6242a3221
created by runtime.gcBgMarkStartWorkers in goroutine 1
	C:/go/src/runtime/mgc.go:1219 +0x1c

goroutine 35 [GC worker (idle), 2 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc00048bf50 sp=0xc00048bf30 pc=0x7ff62427144e
runtime.gcBgMarkWorker()
	C:/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00048bfe0 sp=0xc00048bf50 pc=0x7ff624250645
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00048bfe8 sp=0xc00048bfe0 pc=0x7ff6242a3221
created by runtime.gcBgMarkStartWorkers in goroutine 1
	C:/go/src/runtime/mgc.go:1219 +0x1c

goroutine 7 [GC worker (idle), 2 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc000487f50 sp=0xc000487f30 pc=0x7ff62427144e
runtime.gcBgMarkWorker()
	C:/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000487fe0 sp=0xc000487f50 pc=0x7ff624250645
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000487fe8 sp=0xc000487fe0 pc=0x7ff6242a3221
created by runtime.gcBgMarkStartWorkers in goroutine 1
	C:/go/src/runtime/mgc.go:1219 +0x1c

goroutine 36 [GC worker (idle)]:
runtime.gopark(0x4bf904650d44?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc000493f50 sp=0xc000493f30 pc=0x7ff62427144e
runtime.gcBgMarkWorker()
	C:/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000493fe0 sp=0xc000493f50 pc=0x7ff624250645
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000493fe8 sp=0xc000493fe0 pc=0x7ff6242a3221
created by runtime.gcBgMarkStartWorkers in goroutine 1
	C:/go/src/runtime/mgc.go:1219 +0x1c

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x7ff625f80c00?, 0x1?, 0x4c?, 0xc5?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc00048ff50 sp=0xc00048ff30 pc=0x7ff62427144e
runtime.gcBgMarkWorker()
	C:/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00048ffe0 sp=0xc00048ff50 pc=0x7ff624250645
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00048ffe8 sp=0xc00048ffe0 pc=0x7ff6242a3221
created by runtime.gcBgMarkStartWorkers in goroutine 1
	C:/go/src/runtime/mgc.go:1219 +0x1c

goroutine 37 [GC worker (idle)]:
runtime.gopark(0x4bf9054645e8?, 0x1?, 0x80?, 0xc0?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc000495f50 sp=0xc000495f30 pc=0x7ff62427144e
runtime.gcBgMarkWorker()
	C:/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000495fe0 sp=0xc000495f50 pc=0x7ff624250645
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000495fe8 sp=0xc000495fe0 pc=0x7ff6242a3221
created by runtime.gcBgMarkStartWorkers in goroutine 1
	C:/go/src/runtime/mgc.go:1219 +0x1c

goroutine 9 [GC worker (idle)]:
runtime.gopark(0x4bf90196a140?, 0x1?, 0x68?, 0x5d?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc000491f50 sp=0xc000491f30 pc=0x7ff62427144e
runtime.gcBgMarkWorker()
	C:/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000491fe0 sp=0xc000491f50 pc=0x7ff624250645
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000491fe8 sp=0xc000491fe0 pc=0x7ff6242a3221
created by runtime.gcBgMarkStartWorkers in goroutine 1
	C:/go/src/runtime/mgc.go:1219 +0x1c

goroutine 38 [GC worker (idle)]:
runtime.gopark(0x4bf903455f90?, 0x1?, 0x60?, 0xa3?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc00049bf50 sp=0xc00049bf30 pc=0x7ff62427144e
runtime.gcBgMarkWorker()
	C:/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00049bfe0 sp=0xc00049bf50 pc=0x7ff624250645
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00049bfe8 sp=0xc00049bfe0 pc=0x7ff6242a3221
created by runtime.gcBgMarkStartWorkers in goroutine 1
	C:/go/src/runtime/mgc.go:1219 +0x1c

goroutine 10 [GC worker (idle)]:
runtime.gopark(0x7ff625f80c00?, 0x1?, 0xfc?, 0xfc?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc000497f50 sp=0xc000497f30 pc=0x7ff62427144e
runtime.gcBgMarkWorker()
	C:/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000497fe0 sp=0xc000497f50 pc=0x7ff624250645
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000497fe8 sp=0xc000497fe0 pc=0x7ff6242a3221
created by runtime.gcBgMarkStartWorkers in goroutine 1
	C:/go/src/runtime/mgc.go:1219 +0x1c

goroutine 39 [GC worker (idle)]:
runtime.gopark(0x7ff625f80c00?, 0x1?, 0x9c?, 0x15?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc00049df50 sp=0xc00049df30 pc=0x7ff62427144e
runtime.gcBgMarkWorker()
	C:/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00049dfe0 sp=0xc00049df50 pc=0x7ff624250645
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00049dfe8 sp=0xc00049dfe0 pc=0x7ff6242a3221
created by runtime.gcBgMarkStartWorkers in goroutine 1
	C:/go/src/runtime/mgc.go:1219 +0x1c

goroutine 11 [GC worker (idle)]:
runtime.gopark(0x4bf9054645e8?, 0x1?, 0xe4?, 0xd1?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc000499f50 sp=0xc000499f30 pc=0x7ff62427144e
runtime.gcBgMarkWorker()
	C:/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000499fe0 sp=0xc000499f50 pc=0x7ff624250645
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000499fe8 sp=0xc000499fe0 pc=0x7ff6242a3221
created by runtime.gcBgMarkStartWorkers in goroutine 1
	C:/go/src/runtime/mgc.go:1219 +0x1c

goroutine 40 [GC worker (idle)]:
runtime.gopark(0x7ff625f80c00?, 0x1?, 0x0?, 0xff?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc0004a5f50 sp=0xc0004a5f30 pc=0x7ff62427144e
runtime.gcBgMarkWorker()
	C:/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004a5fe0 sp=0xc0004a5f50 pc=0x7ff624250645
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004a5fe8 sp=0xc0004a5fe0 pc=0x7ff6242a3221
created by runtime.gcBgMarkStartWorkers in goroutine 1
	C:/go/src/runtime/mgc.go:1219 +0x1c

goroutine 20 [GC worker (idle)]:
runtime.gopark(0x4bf905836018?, 0x1?, 0xa4?, 0x53?, 0x0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc000099f50 sp=0xc000099f30 pc=0x7ff62427144e
runtime.gcBgMarkWorker()
	C:/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000099fe0 sp=0xc000099f50 pc=0x7ff624250645
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000099fe8 sp=0xc000099fe0 pc=0x7ff6242a3221
created by runtime.gcBgMarkStartWorkers in goroutine 1
	C:/go/src/runtime/mgc.go:1219 +0x1c

goroutine 12 [chan receive, 2 minutes]:
runtime.gopark(0xc0004a3ea0?, 0x7ff6245968e5?, 0x1?, 0xed?, 0xc0004a3f18?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc0004a3e88 sp=0xc0004a3e68 pc=0x7ff62427144e
runtime.chanrecv(0xc000606060, 0xc0004a3f90, 0x1)
	C:/go/src/runtime/chan.go:583 +0x3d0 fp=0xc0004a3f00 sp=0xc0004a3e88 pc=0x7ff62423adb0
runtime.chanrecv1(0xc000606060?, 0xc000465c20?)
	C:/go/src/runtime/chan.go:442 +0x12 fp=0xc0004a3f28 sp=0xc0004a3f00 pc=0x7ff62423a9b2
main.HandleSignals.func1()
	C:/gopath/src/github.com/rqlite/rqlite/cmd/rqlited/signals.go:20 +0x7d fp=0xc0004a3fe0 sp=0xc0004a3f28 pc=0x7ff624dedc5d
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004a3fe8 sp=0xc0004a3fe0 pc=0x7ff6242a3221
created by main.HandleSignals in goroutine 1
	C:/gopath/src/github.com/rqlite/rqlite/cmd/rqlited/signals.go:16 +0x99

goroutine 50 [syscall, 2 minutes]:
runtime.notetsleepg(0x0?, 0x0?)
	C:/go/src/runtime/lock_sema.go:294 +0x31 fp=0xc00050ffa0 sp=0xc00050ff58 pc=0x7ff624240851
os/signal.signal_recv()
	C:/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc00050ffc0 sp=0xc00050ffa0 pc=0x7ff62429eae9
os/signal.loop()
	C:/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc00050ffe0 sp=0xc00050ffc0 pc=0x7ff624596c13
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00050ffe8 sp=0xc00050ffe0 pc=0x7ff6242a3221
created by os/signal.Notify.func1.1 in goroutine 12
	C:/go/src/os/signal/signal.go:151 +0x1f

goroutine 13 [IO wait, 2 minutes]:
runtime.gopark(0x1f4?, 0x7ff624f5a580?, 0x98?, 0xcc?, 0xc0000accc8?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc0004a7b18 sp=0xc0004a7af8 pc=0x7ff62427144e
runtime.netpollblock(0x17?, 0x242381e6?, 0xf6?)
	C:/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc0004a7b50 sp=0xc0004a7b18 pc=0x7ff624267d97
internal/poll.runtime_pollWait(0x1b7e97912a0, 0x72)
	C:/go/src/runtime/netpoll.go:343 +0x85 fp=0xc0004a7b70 sp=0xc0004a7b50 pc=0x7ff62429ca65
internal/poll.(*pollDesc).wait(0x0?, 0x0?, 0x0)
	C:/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0004a7b98 sp=0xc0004a7b70 pc=0x7ff62431fb27
internal/poll.execIO(0xc0000acc98, 0xc0004a7c38)
	C:/go/src/internal/poll/fd_windows.go:175 +0xe6 fp=0xc0004a7c08 sp=0xc0004a7b98 pc=0x7ff624321046
internal/poll.(*FD).acceptOne(0xc0000acc80, 0xee8, {0xc000eb4000?, 0xc0005ff000?, 0x7ff6252bdae8?}, 0xc0004a7cf0?)
	C:/go/src/internal/poll/fd_windows.go:944 +0x67 fp=0xc0004a7c68 sp=0xc0004a7c08 pc=0x7ff6243256a7
internal/poll.(*FD).Accept(0xc0000acc80, 0xc0004a7e10)
	C:/go/src/internal/poll/fd_windows.go:978 +0x1bc fp=0xc0004a7d20 sp=0xc0004a7c68 pc=0x7ff6243259dc
net.(*netFD).accept(0xc0000acc80)
	C:/go/src/net/fd_windows.go:166 +0x54 fp=0xc0004a7e30 sp=0xc0004a7d20 pc=0x7ff6243753b4
net.(*TCPListener).accept(0xc000117940)
	C:/go/src/net/tcpsock_posix.go:152 +0x1e fp=0xc0004a7e58 sp=0xc0004a7e30 pc=0x7ff62439067e
net.(*TCPListener).Accept(0xc000117940)
	C:/go/src/net/tcpsock.go:315 +0x30 fp=0xc0004a7e88 sp=0xc0004a7e58 pc=0x7ff62438f710
github.com/rqlite/rqlite/v8/tcp.(*Mux).Serve(0xc0000a2820)
	C:/gopath/src/github.com/rqlite/rqlite/tcp/mux.go:150 +0x1ba fp=0xc0004a7fc8 sp=0xc0004a7e88 pc=0x7ff624b8649a
main.startNodeMux.func2()
	C:/gopath/src/github.com/rqlite/rqlite/cmd/rqlited/main.go:389 +0x25 fp=0xc0004a7fe0 sp=0xc0004a7fc8 pc=0x7ff624dea865
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004a7fe8 sp=0xc0004a7fe0 pc=0x7ff6242a3221
created by main.startNodeMux in goroutine 1
	C:/gopath/src/github.com/rqlite/rqlite/cmd/rqlited/main.go:389 +0x6b9

goroutine 14 [chan receive, 2 minutes]:
runtime.gopark(0xc000606120?, 0xc00050bef8?, 0x94?, 0x20?, 0xc000505520?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc00050bec8 sp=0xc00050bea8 pc=0x7ff62427144e
runtime.chanrecv(0xc0001066c0, 0xc00050bf78, 0x1)
	C:/go/src/runtime/chan.go:583 +0x3d0 fp=0xc00050bf40 sp=0xc00050bec8 pc=0x7ff62423adb0
runtime.chanrecv2(0x7ff6242798a0?, 0xc000cc4100?)
	C:/go/src/runtime/chan.go:447 +0x12 fp=0xc00050bf68 sp=0xc00050bf40 pc=0x7ff62423a9d2
github.com/rqlite/rqlite/v8/tcp.(*listener).Accept(0x0?)
	C:/gopath/src/github.com/rqlite/rqlite/tcp/mux.go:251 +0x25 fp=0xc00050bf98 sp=0xc00050bf68 pc=0x7ff624b870c5
github.com/rqlite/rqlite/v8/cluster.(*Service).serve(0xc00047aab0)
	C:/gopath/src/github.com/rqlite/rqlite/cluster/service.go:222 +0x3a fp=0xc00050bfc8 sp=0xc00050bf98 pc=0x7ff624b8f45a
github.com/rqlite/rqlite/v8/cluster.(*Service).Open.func2()
	C:/gopath/src/github.com/rqlite/rqlite/cluster/service.go:159 +0x25 fp=0xc00050bfe0 sp=0xc00050bfc8 pc=0x7ff624b8ec05
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00050bfe8 sp=0xc00050bfe0 pc=0x7ff6242a3221
created by github.com/rqlite/rqlite/v8/cluster.(*Service).Open in goroutine 1
	C:/gopath/src/github.com/rqlite/rqlite/cluster/service.go:159 +0x56

goroutine 15 [select, 2 minutes]:
runtime.gopark(0xc00050df68?, 0x4?, 0x50?, 0xe0?, 0xc00050df00?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc00050dd90 sp=0xc00050dd70 pc=0x7ff62427144e
runtime.selectgo(0xc00050df68, 0xc00050def8, 0x0?, 0x0, 0x0?, 0x1)
	C:/go/src/runtime/select.go:327 +0x725 fp=0xc00050deb0 sp=0xc00050dd90 pc=0x7ff624281685
github.com/rqlite/rqlite/v8/queue.(*Queue[...]).run(0x7ff625697e00)
	C:/gopath/src/github.com/rqlite/rqlite/queue/queue.go:196 +0x17d fp=0xc00050dfc0 sp=0xc00050deb0 pc=0x7ff624de1a7d
github.com/rqlite/rqlite/v8/queue.New[...].func1()
	C:/gopath/src/github.com/rqlite/rqlite/queue/queue.go:116 +0x25 fp=0xc00050dfe0 sp=0xc00050dfc0 pc=0x7ff624de18c5
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00050dfe8 sp=0xc00050dfe0 pc=0x7ff6242a3221
created by github.com/rqlite/rqlite/v8/queue.New[...] in goroutine 1
	C:/gopath/src/github.com/rqlite/rqlite/queue/queue.go:116 +0x213

goroutine 16 [select, 2 minutes]:
runtime.gopark(0xc0004aff30?, 0x2?, 0x0?, 0x0?, 0xc0004afeac?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc0004afd30 sp=0xc0004afd10 pc=0x7ff62427144e
runtime.selectgo(0xc0004aff30, 0xc0004afea8, 0x0?, 0x0, 0x0?, 0x1)
	C:/go/src/runtime/select.go:327 +0x725 fp=0xc0004afe50 sp=0xc0004afd30 pc=0x7ff624281685
github.com/rqlite/rqlite/v8/http.(*Service).runQueue(0xc0001aa240)
	C:/gopath/src/github.com/rqlite/rqlite/http/service.go:1525 +0x125 fp=0xc0004affc8 sp=0xc0004afe50 pc=0x7ff624ddfd85
github.com/rqlite/rqlite/v8/http.(*Service).Start.func5()
	C:/gopath/src/github.com/rqlite/rqlite/http/service.go:409 +0x25 fp=0xc0004affe0 sp=0xc0004affc8 pc=0x7ff624dd7365
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004affe8 sp=0xc0004affe0 pc=0x7ff6242a3221
created by github.com/rqlite/rqlite/v8/http.(*Service).Start in goroutine 1
	C:/gopath/src/github.com/rqlite/rqlite/http/service.go:409 +0x846

goroutine 66 [runnable]:
runtime.gopark(0x208?, 0x7ff624f5a580?, 0x18?, 0xcf?, 0xc0000acf48?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc000509a68 sp=0xc000509a48 pc=0x7ff62427144e
runtime.netpollblock(0x17?, 0x242381e6?, 0xf6?)
	C:/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc000509aa0 sp=0xc000509a68 pc=0x7ff624267d97
internal/poll.runtime_pollWait(0x1b7e97911a8, 0x72)
	C:/go/src/runtime/netpoll.go:343 +0x85 fp=0xc000509ac0 sp=0xc000509aa0 pc=0x7ff62429ca65
internal/poll.(*pollDesc).wait(0x0?, 0x0?, 0x0)
	C:/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000509ae8 sp=0xc000509ac0 pc=0x7ff62431fb27
internal/poll.execIO(0xc0000acf18, 0xc000509b88)
	C:/go/src/internal/poll/fd_windows.go:175 +0xe6 fp=0xc000509b58 sp=0xc000509ae8 pc=0x7ff624321046
internal/poll.(*FD).acceptOne(0xc0000acf00, 0x542c8, {0xc0213b2870?, 0xc02192ec00?, 0x7ff6252bdae8?}, 0xc000509c40?)
	C:/go/src/internal/poll/fd_windows.go:944 +0x67 fp=0xc000509bb8 sp=0xc000509b58 pc=0x7ff6243256a7
internal/poll.(*FD).Accept(0xc0000acf00, 0xc000509d60)
	C:/go/src/internal/poll/fd_windows.go:978 +0x1bc fp=0xc000509c70 sp=0xc000509bb8 pc=0x7ff6243259dc
net.(*netFD).accept(0xc0000acf00)
	C:/go/src/net/fd_windows.go:166 +0x54 fp=0xc000509d80 sp=0xc000509c70 pc=0x7ff6243753b4
net.(*TCPListener).accept(0xc000117980)
	C:/go/src/net/tcpsock_posix.go:152 +0x1e fp=0xc000509da8 sp=0xc000509d80 pc=0x7ff62439067e
net.(*TCPListener).Accept(0xc000117980)
	C:/go/src/net/tcpsock.go:315 +0x30 fp=0xc000509dd8 sp=0xc000509da8 pc=0x7ff62438f710
net/http.(*onceCloseListener).Accept(0xc02191c2d0?)
	<autogenerated>:1 +0x24 fp=0xc000509df0 sp=0xc000509dd8 pc=0x7ff624563aa4
net/http.(*Server).Serve(0xc0001aa240, {0x7ff62568e2a0, 0xc000117980})
	C:/go/src/net/http/server.go:3056 +0x364 fp=0xc000509f20 sp=0xc000509df0 pc=0x7ff624541724
github.com/rqlite/rqlite/v8/http.(*Service).Start.func1()
	C:/gopath/src/github.com/rqlite/rqlite/http/service.go:414 +0x36 fp=0xc000509fe0 sp=0xc000509f20 pc=0x7ff624dd7116
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000509fe8 sp=0xc000509fe0 pc=0x7ff6242a3221
created by github.com/rqlite/rqlite/v8/http.(*Service).Start in goroutine 1
	C:/gopath/src/github.com/rqlite/rqlite/http/service.go:413 +0x9db

goroutine 67 [chan receive, 2 minutes]:
runtime.gopark(0xc031313535?, 0xc0009181b0?, 0xc?, 0x0?, 0x7ff62567d9b0?)
	C:/go/src/runtime/proc.go:398 +0xce fp=0xc0004abe00 sp=0xc0004abde0 pc=0x7ff62427144e
runtime.chanrecv(0xc000106600, 0xc0004abeb0, 0x1)
	C:/go/src/runtime/chan.go:583 +0x3d0 fp=0xc0004abe78 sp=0xc0004abe00 pc=0x7ff62423adb0
runtime.chanrecv2(0x1b7a4232298?, 0x30?)
	C:/go/src/runtime/chan.go:447 +0x12 fp=0xc0004abea0 sp=0xc0004abe78 pc=0x7ff62423a9d2
github.com/rqlite/rqlite/v8/tcp.(*listener).Accept(0x7ff6242798a0?)
	C:/gopath/src/github.com/rqlite/rqlite/tcp/mux.go:251 +0x25 fp=0xc0004abed0 sp=0xc0004abea0 pc=0x7ff624b870c5
github.com/rqlite/rqlite/v8/tcp.(*Layer).Accept(0x7ff624c16b5b?)
	C:/gopath/src/github.com/rqlite/rqlite/tcp/mux.go:61 +0x1b fp=0xc0004abee8 sp=0xc0004abed0 pc=0x7ff624b85ebb
github.com/rqlite/rqlite/v8/store.(*Transport).Accept(0xc000944048?)
	C:/gopath/src/github.com/rqlite/rqlite/store/transport.go:39 +0x1b fp=0xc0004abf00 sp=0xc0004abee8 pc=0x7ff624d74c7b
github.com/hashicorp/raft.(*NetworkTransport).listen(0xc0001d4000)
	c:/gopath/pkg/mod/github.com/hashicorp/[email protected]/net_transport.go:561 +0x47 fp=0xc0004abfc8 sp=0xc0004abf00 pc=0x7ff624c167e7
github.com/hashicorp/raft.NewNetworkTransportWithConfig.func1()
	c:/gopath/pkg/mod/github.com/hashicorp/[email protected]/net_transport.go:240 +0x25 fp=0xc0004abfe0 sp=0xc0004abfc8 pc=0x7ff624c149a5
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004abfe8 sp=0xc0004abfe0 pc=0x7ff6242a3221
created by github.com/hashicorp/raft.NewNetworkTransportWithConfig in goroutine 1
	c:/gopath/pkg/mod/github.com/hashicorp/[email protected]/net_transport.go:240 +0x2a5

goroutine 16474 [syscall, locked to thread]:
runtime.cgocall(0x7ff6242a49e0, 0xc004c7fac8)
	C:/go/src/runtime/cgocall.go:157 +0x3e fp=0xc001405108 sp=0xc0014050d0 pc=0x7ff6242386de
syscall.SyscallN(0x7ffbfe8801e0?, {0xc0014051a0?, 0x3?, 0x0?})
	C:/go/src/runtime/syscall_windows.go:544 +0x107 fp=0xc001405180 sp=0xc001405108 pc=0x7ff62429ff07
syscall.Syscall(0x0?, 0xc004c7f800?, 0x7ff6252bdae8?, 0xc001405228?, 0x7ff62429fc55?)
	C:/go/src/runtime/syscall_windows.go:482 +0x35 fp=0xc0014051c8 sp=0xc001405180 pc=0x7ff62429fc55
syscall.CloseHandle(0xb464?)
	C:/go/src/syscall/zsyscall_windows.go:482 +0x52 fp=0xc001405210 sp=0xc0014051c8 pc=0x7ff6242fb412
internal/poll.(*FD).destroy(0xc01e725900)
	C:/go/src/internal/poll/fd_windows.go:372 +0x74 fp=0xc001405238 sp=0xc001405210 pc=0x7ff6243218d4
internal/poll.(*FD).decref(0xc0202a40c0?)
	C:/go/src/internal/poll/fd_mutex.go:213 +0x53 fp=0xc001405258 sp=0xc001405238 pc=0x7ff62431f813
internal/poll.(*FD).Close(0xc01e725900)
	C:/go/src/internal/poll/fd_windows.go:390 +0x69 fp=0xc001405288 sp=0xc001405258 pc=0x7ff6243219a9
os.(*file).close(0xc01e725900)
	C:/go/src/os/file_windows.go:203 +0x69 fp=0xc0014052d8 sp=0xc001405288 pc=0x7ff62432fbc9
os.(*File).Close(0x1a401405301?)
	C:/go/src/os/file_posix.go:23 +0x1b fp=0xc0014052f0 sp=0xc0014052d8 pc=0x7ff62432eb1b
github.com/rqlite/rqlite/v8/db.lastModified.func2()
	C:/gopath/src/github.com/rqlite/rqlite/db/db.go:1543 +0x25 fp=0xc001405308 sp=0xc0014052f0 pc=0x7ff624bcd385
github.com/rqlite/rqlite/v8/db.lastModified({0xc00013e5d0, 0x2d})
	C:/gopath/src/github.com/rqlite/rqlite/db/db.go:1551 +0x168 fp=0xc001405398 sp=0xc001405308 pc=0x7ff624bcd248
github.com/rqlite/rqlite/v8/db.(*DB).WALLastModified(...)
	C:/gopath/src/github.com/rqlite/rqlite/db/db.go:227
github.com/rqlite/rqlite/v8/db.(*DB).LastModified(0xc0000a0900)
	C:/gopath/src/github.com/rqlite/rqlite/db/db.go:205 +0x52 fp=0xc001405408 sp=0xc001405398 pc=0x7ff624bc2bb2
github.com/rqlite/rqlite/v8/db.(*DB).Stats(0xc0000a0900)
	C:/gopath/src/github.com/rqlite/rqlite/db/db.go:277 +0x648 fp=0xc001405548 sp=0xc001405408 pc=0x7ff624bc3408
github.com/rqlite/rqlite/v8/db.(*SwappableDB).Stats(0x7ff625176440?)
	C:/gopath/src/github.com/rqlite/rqlite/db/swappable_db.go:67 +0x89 fp=0xc0014055a8 sp=0xc001405548 pc=0x7ff624bce469
github.com/rqlite/rqlite/v8/store.(*Store).Stats(0xc000004300)
	C:/gopath/src/github.com/rqlite/rqlite/store/store.go:1003 +0x45 fp=0xc0014059e0 sp=0xc0014055a8 pc=0x7ff624d678a5
github.com/rqlite/rqlite/v8/http.(*Service).handleStatus(0xc0001aa240, {0x7ff62568ecb0, 0xc00f83c700}, 0xc01194e800, 0xc001405b68?)
	C:/gopath/src/github.com/rqlite/rqlite/http/service.go:794 +0x18c fp=0xc001405af8 sp=0xc0014059e0 pc=0x7ff624dda08c
github.com/rqlite/rqlite/v8/http.(*Service).ServeHTTP(0x7ff625f7efe0?, {0x7ff62568ecb0, 0xc00f83c700}, 0xc01194e800)
	C:/gopath/src/github.com/rqlite/rqlite/http/service.go:486 +0x469 fp=0xc001405b48 sp=0xc001405af8 pc=0x7ff624dd7aa9
net/http.serverHandler.ServeHTTP({0xc00fd7ee70?}, {0x7ff62568ecb0?, 0xc00f83c700?}, 0x6?)
	C:/go/src/net/http/server.go:2938 +0x8e fp=0xc001405b78 sp=0xc001405b48 pc=0x7ff6245412ce
net/http.(*conn).serve(0xc007f27200, {0x7ff625692528, 0xc0001d61b0})
	C:/go/src/net/http/server.go:2009 +0x5f4 fp=0xc001405fb8 sp=0xc001405b78 pc=0x7ff62453d1b4
net/http.(*Server).Serve.func3()
	C:/go/src/net/http/server.go:3086 +0x28 fp=0xc001405fe0 sp=0xc001405fb8 pc=0x7ff624541ae8
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc001405fe8 sp=0xc001405fe0 pc=0x7ff6242a3221
created by net/http.(*Server).Serve in goroutine 66
	C:/go/src/net/http/server.go:3086 +0x5cb

goroutine 6013 [syscall, locked to thread]:
runtime.cgocall(0x7ff6242a49e0, 0xc00467f2c8)
	C:/go/src/runtime/cgocall.go:157 +0x3e fp=0xc007102f98 sp=0xc007102f60 pc=0x7ff6242386de
syscall.SyscallN(0x7ffbfe880650?, {0xc007103030?, 0x3?, 0x0?})
	C:/go/src/runtime/syscall_windows.go:544 +0x107 fp=0xc007103010 sp=0xc007102f98 pc=0x7ff62429ff07
syscall.Syscall(0x29?, 0xc01c1650e0?, 0xffffffffffffffff?, 0x0?, 0x0?)
	C:/go/src/runtime/syscall_windows.go:482 +0x35 fp=0xc007103058 sp=0xc007103010 pc=0x7ff62429fc55
syscall.GetFileAttributesEx(0xc01f9687c0?, 0x0, 0x0?)
	C:/go/src/syscall/zsyscall_windows.go:750 +0x65 fp=0xc0071030a8 sp=0xc007103058 pc=0x7ff6242fcb25
os.stat({0x7ff62523ff3e, 0x5}, {0xc01f9687c0, 0x29}, 0x0)
	C:/go/src/os/stat_windows.go:35 +0x289 fp=0xc0071033f8 sp=0xc0071030a8 pc=0x7ff6243317e9
os.lstatNolog({0xc01f9687c0?, 0x40?})
	C:/go/src/os/stat_windows.go:135 +0x4c fp=0xc007103430 sp=0xc0071033f8 pc=0x7ff62433222c
os.Lstat({0xc01f9687c0, 0x29})
	C:/go/src/os/stat.go:22 +0x2c fp=0xc007103450 sp=0xc007103430 pc=0x7ff6243314cc
path/filepath.walk({0xc00003ca80, 0x1f}, {0x7ff625694290, 0xc017055ce0}, 0xc007103588)
	C:/go/src/path/filepath/path.go:495 +0x1ab fp=0xc007103518 sp=0xc007103450 pc=0x7ff62439d6eb
path/filepath.Walk({0xc00003ca80, 0x1f}, 0xc007103588)
	C:/go/src/path/filepath/path.go:572 +0x66 fp=0xc007103568 sp=0xc007103518 pc=0x7ff62439d8e6
github.com/rqlite/rqlite/v8/store.dirSize({0xc00003ca80?, 0xc019e8d5f0?})
	C:/gopath/src/github.com/rqlite/rqlite/store/store.go:2379 +0x45 fp=0xc0071035a8 sp=0xc007103568 pc=0x7ff624d74a25
github.com/rqlite/rqlite/v8/store.(*Store).Stats(0xc000004300)
	C:/gopath/src/github.com/rqlite/rqlite/store/store.go:1035 +0x791 fp=0xc0071039e0 sp=0xc0071035a8 pc=0x7ff624d67ff1
github.com/rqlite/rqlite/v8/http.(*Service).handleStatus(0xc0001aa240, {0x7ff62568ecb0, 0xc008169a40}, 0xc000a28300, 0xc007103b68?)
	C:/gopath/src/github.com/rqlite/rqlite/http/service.go:794 +0x18c fp=0xc007103af8 sp=0xc0071039e0 pc=0x7ff624dda08c
github.com/rqlite/rqlite/v8/http.(*Service).ServeHTTP(0x7ff625f7efe0?, {0x7ff62568ecb0, 0xc008169a40}, 0xc000a28300)
	C:/gopath/src/github.com/rqlite/rqlite/http/service.go:486 +0x469 fp=0xc007103b48 sp=0xc007103af8 pc=0x7ff624dd7aa9
net/http.serverHandler.ServeHTTP({0xc006397080?}, {0x7ff62568ecb0?, 0xc008169a40?}, 0x6?)
	C:/go/src/net/http/server.go:2938 +0x8e fp=0xc007103b78 sp=0xc007103b48 pc=0x7ff6245412ce
net/http.(*conn).serve(0xc005013b90, {0x7ff625692528, 0xc0001d61b0})
	C:/go/src/net/http/server.go:2009 +0x5f4 fp=0xc007103fb8 sp=0xc007103b78 pc=0x7ff62453d1b4
net/http.(*Server).Serve.func3()
	C:/go/src/net/http/server.go:3086 +0x28 fp=0xc007103fe0 sp=0xc007103fb8 pc=0x7ff624541ae8
runtime.goexit()
	C:/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc007103fe8 sp=0xc007103fe0 pc=0x7ff6242a3221
created by net/http.(*Server).Serve in goroutine 66
	C:/go/src/net/http/server.go:3086 +0x5cb

from rqlite.

otoolep commented on May 27, 2024

What stands out is that the handler for /status fails to write its respond to the client. This is surprising, I should log this, instead of calling http.Error. That said, I don't see how it could be related.

Something is clearly consuming resources at a high rate, and not releasing them. I haven't seen this before, and am keen to fix it.

from rqlite.

otoolep commented on May 27, 2024

@mmolder -- I presume you are building your own container images, since I don't supply a Windows one. Do you think you can build and run HEAD? There is one improvement to logging which I'd like to see execute -- it will tell me which error is actually happening which triggers this message:

[rqlited] 2024/04/03 14:16:26 http: superfluous response.WriteHeader call from github.com/rqlite/rqlite/v8/http.(*Service).handleStatus (service.go:908)

from rqlite.

mmolder commented on May 27, 2024

@otoolep I am running the latest windows build available here v8.23.0. Is this a new one or still building?

from rqlite.

otoolep commented on May 27, 2024

Use this one, which just built with my latest changes: https://ci.appveyor.com/project/otoolep/rqlite/build/artifacts

See the CHANGELOG for more details.

from rqlite.

otoolep commented on May 27, 2024

https://github.com/rqlite/rqlite/blob/master/CHANGELOG.md

from rqlite.

mmolder commented on May 27, 2024

Copy, I will try it out tomorrow

from rqlite.

mmolder commented on May 27, 2024

I found the issue of my problem. It was me accidentally spamming requests to the /status endpoint to check if a server was running rqlite in order to join it and find its database. When starting and stopping the rqlite process though a service I am currently developing, I am polling my servers to see if there already exists a running cluster on the network using the /status and /nodes endpoints. A mistake in my code spawned more and more threads which all queried the /status endpoint several times every few seconds which lead to the high traffic and in turn spiked the CPU usage. This was partly triggered by me pulling the network cables and having the process restart. After fixing it, things seem to work as expected.

from rqlite.

otoolep commented on May 27, 2024

OK, I was wondering if it might be something that like that. Hitting the /status endpoint does reach into all layers of rqlite (store, database, network) etc. as its goal is to provide as much information as possible about the node. So it does involve a fair number of code paths.

That said, you must have been really hammering it. But I would prefer not have to have rqlite fall over simply because of this particular issue in your code. I'd have to add some sort of rate-limiting to protect the system -- but I'm not convinced it's quite worth it either.

I'll close this issue for now, as there are higher-priority issues I need to work on. I'm glad you got to the root cause, and you did expose a minor bug in the code.

from rqlite.

otoolep commented on May 27, 2024

You might prefer to use the /readyz endpoint, perhaps with the noleader flag. That is a very lightweight check, and will tell you what you need to know, but at very low cost.

See https://rqlite.io/docs/guides/monitoring-rqlite/#readiness-checks

from rqlite.

mmolder commented on May 27, 2024

Thank you for all the help! I was looking at using /readyz but it does not contain the database name of the running node I'm querying. Is there any lightweight check to find this other than with /status? This is really all I need to know other than what /readyz is providing.

from rqlite.

otoolep commented on May 27, 2024

What do you mean by "database name"? Do you mean the node ID?

from rqlite.

mmolder commented on May 27, 2024

I am interesting in getting the storage location i.e. what you get from the /status response in store/dir.

from rqlite.

otoolep commented on May 27, 2024

Can you show me precisely which key in the /status response you are talking about?

from rqlite.

mmolder commented on May 27, 2024

This:

from rqlite.

otoolep commented on May 27, 2024

Let's take a step back. What are you doing with that information?

That path is simply the path to the data directory path you pass to rqlited when you launch rqlited in the first place. You know it since you launch rqlited.

from rqlite.

mmolder commented on May 27, 2024

I am automating the process of starting and stopping nodes in a cluster as part of a bigger service. When a node first is started I supply it with the data directory path (C:\tmp\dbname in the example above) which I want to be the same for all nodes joining the cluster for consistency. For the next node joining the cluster I scan my local network for servers already running a rqlite process and if found join it and use the same local storage path. They are all connected on a specific subnet.

from rqlite.

rqlited + DNS client 100% CPU usage after network disconnecting (windows) about rqlite HOT 21 CLOSED

Comments (21)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent