[BUG] yolov4 tiny example produces fatal error: unexpected signal during runtime execution about go-darknet HOT 5 CLOSED

Hi!
I'll take a look soon

from go-darknet.

stacktrace:

`go build -o base_example/main base_example/main.go && ./base_example/main --configFile=yolov4-tiny.cfg --weightsFile=yolov4-tiny.weights --imageFile=sample.jpg
go: downloading github.com/edsrzf/mmap-go v1.1.0
Try to load cfg: yolov4-tiny.cfg, clear = 0
0 : compute_capability = 860, cudnn_half = 0, GPU: NVIDIA GeForce RTX 3090
net.optimized_memory = 0
mini_batch = 64, batch = 64, time_steps = 1, train = 1
layer filters size/strd(dil) input output
0 Create CUDA-stream - 0
Create cudnn-handle 0
conv 32 3 x 3/ 2 416 x 416 x 3 -> 208 x 208 x 32 0.075 BF
1 conv 64 3 x 3/ 2 208 x 208 x 32 -> 104 x 104 x 64 0.399 BF
2 conv 64 3 x 3/ 1 104 x 104 x 64 -> 104 x 104 x 64 0.797 BF
3 route 2 1/2 -> 104 x 104 x 32
4 conv 32 3 x 3/ 1 104 x 104 x 32 -> 104 x 104 x 32 0.199 BF
5 conv 32 3 x 3/ 1 104 x 104 x 32 -> 104 x 104 x 32 0.199 BF
6 route 5 4 -> 104 x 104 x 64
7 conv 64 1 x 1/ 1 104 x 104 x 64 -> 104 x 104 x 64 0.089 BF
8 route 2 7 -> 104 x 104 x 128
9 max 2x 2/ 2 104 x 104 x 128 -> 52 x 52 x 128 0.001 BF
10 conv 128 3 x 3/ 1 52 x 52 x 128 -> 52 x 52 x 128 0.797 BF
11 route 10 1/2 -> 52 x 52 x 64
12 conv 64 3 x 3/ 1 52 x 52 x 64 -> 52 x 52 x 64 0.199 BF
13 conv 64 3 x 3/ 1 52 x 52 x 64 -> 52 x 52 x 64 0.199 BF
14 route 13 12 -> 52 x 52 x 128
15 conv 128 1 x 1/ 1 52 x 52 x 128 -> 52 x 52 x 128 0.089 BF
16 route 10 15 -> 52 x 52 x 256
17 max 2x 2/ 2 52 x 52 x 256 -> 26 x 26 x 256 0.001 BF
18 conv 256 3 x 3/ 1 26 x 26 x 256 -> 26 x 26 x 256 0.797 BF
19 route 18 1/2 -> 26 x 26 x 128
20 conv 128 3 x 3/ 1 26 x 26 x 128 -> 26 x 26 x 128 0.199 BF
21 conv 128 3 x 3/ 1 26 x 26 x 128 -> 26 x 26 x 128 0.199 BF
22 route 21 20 -> 26 x 26 x 256
23 conv 256 1 x 1/ 1 26 x 26 x 256 -> 26 x 26 x 256 0.089 BF
24 route 18 23 -> 26 x 26 x 512
25 max 2x 2/ 2 26 x 26 x 512 -> 13 x 13 x 512 0.000 BF
26 conv 512 3 x 3/ 1 13 x 13 x 512 -> 13 x 13 x 512 0.797 BF
27 conv 256 1 x 1/ 1 13 x 13 x 512 -> 13 x 13 x 256 0.044 BF
28 conv 512 3 x 3/ 1 13 x 13 x 256 -> 13 x 13 x 512 0.399 BF
29 conv 255 1 x 1/ 1 13 x 13 x 512 -> 13 x 13 x 255 0.044 BF
30 yolo
[yolo] params: iou loss: ciou (4), iou_norm: 0.07, obj_norm: 1.00, cls_norm: 1.00, delta_norm: 1.00, scale_x_y: 1.05
nms_kind: greedynms (1), beta = 0.600000
31 route 27 -> 13 x 13 x 256
32 conv 128 1 x 1/ 1 13 x 13 x 256 -> 13 x 13 x 128 0.011 BF
33 upsample 2x 13 x 13 x 128 -> 26 x 26 x 128
34 route 33 23 -> 26 x 26 x 384
35 conv 256 3 x 3/ 1 26 x 26 x 384 -> 26 x 26 x 256 1.196 BF
36 conv 255 1 x 1/ 1 26 x 26 x 256 -> 26 x 26 x 255 0.088 BF
37 yolo
[yolo] params: iou loss: ciou (4), iou_norm: 0.07, obj_norm: 1.00, cls_norm: 1.00, delta_norm: 1.00, scale_x_y: 1.05
nms_kind: greedynms (1), beta = 0.600000
Unused field: 'names = coco.names'
Total BFLOPS 6.910
avg_outputs = 310203
Allocate additional workspace_size = 535.18 MB
Try to load weights: yolov4-tiny.weights
Loading weights from yolov4-tiny.weights...
seen 64, trained: 0 K-images (0 Kilo-batches_64)
Done! Loaded 38 layers from weights-file
Loaded - names_list: coco.names, classes = 80
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x797c8340 pc=0x7f9e7a6eddb0]

runtime stack:
runtime.throw({0x506543?, 0x0?})
/usr/lib/go/src/runtime/panic.go:1047 +0x5d fp=0x7ffc3bd4ca78 sp=0x7ffc3bd4ca48 pc=0x4353bd
runtime.sigpanic()
/usr/lib/go/src/runtime/signal_unix.go:819 +0x369 fp=0x7ffc3bd4cac8 sp=0x7ffc3bd4ca78 pc=0x449529

goroutine 1 [syscall]:
runtime.cgocall(0x4d1360, 0xc000085b98)
/usr/lib/go/src/runtime/cgocall.go:158 +0x5c fp=0xc000085b70 sp=0xc000085b38 pc=0x40565c
github.com/LdDl/go-darknet._Cfunc_perform_network_detect(0x2652500, 0xc000126010, 0x50, 0x3e800000, 0x3f000000, 0x3ee66666, 0x0)
_cgo_gotypes.go:920 +0x4e fp=0xc000085b98 sp=0xc000085b70 pc=0x4b434e
github.com/LdDl/go-darknet.(*YOLONetwork).Detect.func1(0x219?, 0xc000126000)
/tmp/go-darknet/network.go:151 +0xe9 fp=0xc000085c18 sp=0xc000085b98 pc=0x4b5e69
github.com/LdDl/go-darknet.(*YOLONetwork).Detect(0xc000085e30, 0xc000120080?)
`

from go-darknet.

@Evert-Arends
I do confirm fatal error.

It happens for me on CUDA-based Darknet installation, but not when it just CPU (make install_darknet)

Need more time to investigate and play with different CUDA versions.
Can you test CPU-version to confirm that there is GPU problem only?

p.s. My current CUDA is 11.7. The first thing I'll do: re-install CUDA / downgrade version

Fri Feb  3 11:56:34 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.86.01    Driver Version: 515.86.01    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
|  0%   46C    P8    14W / 170W |    917MiB / 12288MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

from go-darknet.

1. I've downgraded NVidia driver
   

2. Fully removed CUDA/cuDNN

3. Installed CUDA 11.7.0 (notice: NOT v11.7.1) without driver updates by disabling such option in *.run file, since I do not prefer to install CUDA with package manager
   

4. Installed cuDNN: cudnn-linux-x86_64-8.6.0.163

Now it works:

Can you make same experiment?

from go-darknet.

I'll have a look tonight / saturday, I do get my cuda from a package manager, but I can easily change that I suppose. Thanks for responding this quickly!

from go-darknet.