Getting only TPs about confusion-matrix-for-mask-r-cnn HOT 12 CLOSED

I will make the necessary changes and post results shortly. Best

from confusion-matrix-for-mask-r-cnn.

Hi @mehboobali98 , Thets strange, can you please show me what did you pass to the function entries ?

from confusion-matrix-for-mask-r-cnn.

Thanks for swift response, Altimis.

Basically, I'm following the code given in your notebook.

Using code in below image to calculate mAP, gt_tot and pred_tot.

Then, the code in below in image for plotting confusion matrix as you have done in your notebook.

Waiting for your response. Best

from confusion-matrix-for-mask-r-cnn.

Hi again @mehboobali98. first, can you please change the last line by this : tp, fp, fn = utils.plot_confusion_matrix_from_data(gt_tot, pred_tot, dataset.class_names, fz=18, figsize=(20,20), lw=0.5. because i discovred that I did not update the code from the last time to show the class names.
So I suspect that the problem is at the level of the iou_tresh.. can u please add these two lines in the function so that you can see the predicted and ground truth classes being added to the lists : print("Ground truth object : "+dataset.class_names[gt]) and print("Predicted object : "+dataset.class_names[pred]) and show me what's being printed. I know its exausting but there is no other way to know where is the problem...

from confusion-matrix-for-mask-r-cnn.

@mehboobali98 Also, can you change the tresh by 0.5 so that we eliminate this possibility ?

from confusion-matrix-for-mask-r-cnn.

print("Ground truth object : "+dataset.class_names[gt]) and print("Predicted object : "+dataset.class_names[pred]).

After adding the above lines, I'm getting this error:

Without these, the following confusion matrix is generated with IOU = 0.5:

Waiting for your response. Best,

from confusion-matrix-for-mask-r-cnn.

Sorry my bad, try this : print("Ground truth objects : ",gt) and print("Predicted objects : ",pred].
Anyways I think that there is something tricky in your dataset. If the model is able to detect all the classes by 100% precision and recall, the background class shouldnt exist in GT and should be detected 0 times, because this class exists only if the model doesnt detect an object that exists in the imge or if it detects an object that doenst exist in the image :

    for i, gt_class in enumerate(gt_class_ids_["gt_class_ids"]):
        if gt_class_ids_['state'][i] == 0:
            gt.append(gt_class)
            pred.append(0)

#look for objects that are mispredicted (pred objects that dont exists in gt objects)
    for j, pred_class in enumerate(pred_class_ids_["pred_class_ids"]):
        if pred_class_ids_['state'][j] == 0:
            gt.append(0)
            pred.append(pred_class)

State = 0 means that the Gt object or Detected object is not assiciated to any other Detected object or Gt object (so I associate a background class to them to be able to compute the confusion matrix.

In your case, all classes are being detected perfectly, nevertheless there are 2 backgrounds in the GT list and 2 background in the Pred list.

from confusion-matrix-for-mask-r-cnn.

@mehboobali98 Hi again, I tried to do a manual example to check if the gt_pred_list function works.
Check notebook to see the example I made. I also tried it on real-world data but unfortunatly I can't show the results here since the data is confidential.

from confusion-matrix-for-mask-r-cnn.

@Altimis Hello. I've tested a few things.
The gt vector returned from the two functions below two functions should be the same, right?

1. modellib.load_image_gt(test_set, config, image_id, use_mini_mask=False),
2. utils.gt_pred_lists(gt_class_id, gt_bbox, r['class_ids'], r['rois'], 0.5)

Both are giving the gt class ids. However, function 2 is adding class_id=0 to the gt vector for some reason as shown in the images below:

The rest of the gt vectors are the same. Waiting for your response. Best,

from confusion-matrix-for-mask-r-cnn.

Hi Altamis,

After further digging, I have found out that some times my model predicts multiple bboxes (masks) for the same class_id due to which the gt and pred lists are of different lengths. The pred list contains duplicates class_ids in this scenario.

Does your utils.gt_pred_lists(gt_class_id, gt_bbox, r['class_ids'], r['rois'], 0.5) function, handles this case?

from confusion-matrix-for-mask-r-cnn.

@mehboobali98 Hi again. In fact, the modellib.load_image_gt(test_set, config, image_id, use_mini_mask=False) function returns a vector of predicted object that has a different size from the actual GT vector, and with a different order (that is because the order is based on the confidence score). The problem with that is, to plot a confusion matrix, we need to have two vectors (GT and Predictions) that have the same size and with the same order, so that we know if there is correspondance between each index.
The objective of utils.gt_pred_lists(gt_class_id, gt_bbox, r['class_ids'], r['rois'], 0.5) function is to review each predicted object in the original GT vector (where there is no background class) from the first function and do a correspondance with the original predicted vector and order them (as u can see in the 2 lists returned by the second function). For each class in the GT vector that has no correspondent, I associate a background class to it, indicating that this class exists in the image but the model predicted a background class instead, and vice versa. Then after you browse all the images in your test dataset you get two equal size vectors (gt_tot and pred_tot) , where for each index in the gt_tot, you find its correspondent in the same index in pred_tot.

I hope you get the idea behind this solution that I found very helpful especially that I didnt find any other solution when I wanted to plot the confusion matrix and get the TPs FPs and FNs for each class, like a classic classification problem.

from confusion-matrix-for-mask-r-cnn.

@mehboobali98 The gt and pred lists are made so that they have the same size. Even if you have multiple bboxes for the same classes (which means that in the same image there are multiple objects of the same class, which is normal).

from confusion-matrix-for-mask-r-cnn.