A tiny tool to inspect your go process's goroutine info, without single line change in your code. Inspired by py-spy, learned a lot from delve

sudo ./gospy summary --pid 1234

aggregate goroutines by functions: sudo ./gospy top --pid 1234

dump a global variable value by name: sudo ./gospy var --pid 1234 --name runtime.ncpu

type: int32, value: 6

Support types:

• int(8, 16, 32, 64)
• uint(8, 16, 32, 64)
• bool
• string

dump runtime heap info: sudo ./gospy heap --pid 1234

PagesInUse: 918, PagesSwept: 977
Large Object(>32KB) Stats: AllocNum: 21, AllocRamSize: 4.52MB, FreeNum: 16, FreedRamSize: 3.38MB
SweepDone: 1, Sweepers: 0, Sweepgen: 82
P0, FlushGen:82:
        Tiny size object(<16B): AllocNum: 9, BytesUsage: 11/16
        Large size object freed(>32KB): FreeNum: 0, FreedRamSize: 0B
        Small size object(<32KB):
                0B~8B: npages: 1, allocCount: 114
                8B~16B: npages: 1, allocCount: 259
                16B~32B: npages: 1, allocCount: 146
                32B~48B: npages: 1, allocCount: 21
                48B~64B: npages: 1, allocCount: 40
                64B~80B: npages: 1, allocCount: 88
                80B~96B: npages: 1, allocCount: 76
                96B~112B: npages: 1, allocCount: 17
                112B~128B: npages: 1, allocCount: 48
                144B~160B: npages: 1, allocCount: 26
                160B~176B: npages: 1, allocCount: 13
                176B~192B: npages: 1, allocCount: 14
                208B~224B: npages: 1, allocCount: 24
                240B~256B: npages: 1, allocCount: 19
                256B~288B: npages: 1, allocCount: 15
                288B~320B: npages: 1, allocCount: 16
                352B~384B: npages: 1, allocCount: 19
                480B~512B: npages: 1, allocCount: 14
                512B~576B: npages: 1, allocCount: 1
                576B~640B: npages: 1, allocCount: 5
                704B~768B: npages: 1, allocCount: 6
                768B~896B: npages: 1, allocCount: 7
                896B~1.00KB: npages: 1, allocCount: 2
                1.00KB~1.12KB: npages: 1, allocCount: 4
                1.12KB~1.25KB: npages: 1, allocCount: 3
                1.25KB~1.38KB: npages: 2, allocCount: 8
                2.00KB~2.25KB: npages: 2, allocCount: 5
                4.75KB~5.25KB: npages: 2, allocCount: 3
P1, FlushGen: 82:
    ...

Works on target go version >= 1.21 tested:

• 1.21.x

Minium go version to build: 1.13.0

git clone https://github.com/monsterxx03/gospy.git

gospy is based on go module, please don't put repo under GOPATH, it won't work.

cd gospy && make

binary will be created under gospy/bin/

• x86_64 linux only
• Don't work with binaries without debug info, if you build with linker flag -w -s, gospy won't be able to figure out function/variable names.

If binary is build with -buildmode=pie, only support go >= 1.15. Since .gopclntab is missing in PIE mode before 1.15.

Read DWARF ino from ELF binary(embed by go compiler), parse some basic global variables'(runtime.allgs, runtime.allglen...) virtual memory address, then read target process's memory space to recreate runtime structs(runtime.g, runtime.p, runtime.m, runtime.sched...)

There're three ways:

• PTRACE_PEEKDATA, it can only read a long word(8 bytes) at a time. If we want to read a continuous memory space, need to call it multi times.
• process_vm_readv, available after kernel 3.2, can read a continuous block of process memory space, not exposed directly in go, maybe need cgo to call?
• Read /proc/{pid}/mem directly, it's the easiest way on linux. Also more efficient than a syscall in go.

gospy takes the third way(/proc/{pid}/mem). Bad side is sudo privilege is required.

Yes. By default, gospy use PTRACE_ATTACH to suspend target process to get a consistent memory view, after reading, PTRACE_DEATCH to resume target process.

If --non-blocking option is provided, gospy will do memory reading directly, won't suspend target process. If target process is creating/destorying goroutines actively, it may fail during reading memory.

You can compile a binary with debug info and specify the --bin option. Ensure compile with same code revision, same go version.

Yes :)

• kubectl cp bin/gospy pod-xxx:/tmp/gospy
• kubectl exec -it pod-xxx -- /tmp/gospy top --pid 1

go get github.com/monsterxx03/gospy

import (
    "fmt"
    "github.com/monsterxx03/gospy/pkg/proc"
)

func main() {
    p, _ := proc.New(14791, "")
    ver, _ := p.GoVersion()
    fmt.Println(ver) // print target process go version
    sum, _ := p.Summary(true)
    fmt.Println(sum)
    for _, g := range sum.Gs {
        fmt.Println(g)
    }
}

• Support dump more variable types
• Optimize performance