will WBF be better than nms in a single model? about weighted-boxes-fusion HOT 4 OPEN

I test this algorithm on a single faster rcnn and get a worse performance than nms.

So, I posit, the algorithm has certain limitations

from weighted-boxes-fusion.

Just to be sure about performance on single model, I made an experiment.

I took RetinaNet based on ResNet152 backbone which was trained on Open Images dataset. Then I cut final NMS layer. With this model I predicted TOP 500 raw boxes based on confidence for each validation image. Then I calculated metric mAP(0.5):

1. No NMS (Raw boxes) – mAP: 0.171762 – value is small because of large amount of intersected boxes at the same object.
2. With NMS and default THR - mAP: 0.490199 IOU THR: 0.5
3. With NMS and best THR - mAP: 0.490588 IOU THR: 0.47
4. With WBF and optimal parameters (grid search): mAP: 0.453182 IOU THR: 0.43 Skip box THR: 0.21

So, for this case WBF is worse than NMS for single model. Probably WBF (in current implementation) is bad for case of large amount of boxes with poor prediction quality.

from weighted-boxes-fusion.

I didn't have chance to check it. All my models had NMS block at the end. I still have plans to make some related experiments.

from weighted-boxes-fusion.

@ZFTurbo I am using YOLOv8n, YOLOv8s, YOLOv8m, YOLOv8l, YOLOv8x models with WBF for object detection in VisDrone2019 dataset. It seems like the confidence scores increase when used with WBF but the number of prediction decreases.

I would like to know how can I calculate or perform evaluation of the ensemble model predictions in terms of mAP, precision, recall? Is there any bultin function provided in the WBF repository to perform the ensemble model evaluation?

I appreciate your help.

Regards,
Bijay

from weighted-boxes-fusion.