How can this code be used to compute the matrix and values for the entire dataset. For example, I used this code:

def confusion_matrix(dataset, model, cfg):
gt_tot = np.array([])
pred_tot = np.array([])
# mAP list
mAP_ = []

for image_id in dataset.image_ids:
    # load image, bounding boxes and masks for the image id
    image, image_meta, gt_class_id, gt_bbox, gt_mask = modellib.load_image_gt(dataset, cfg, image_id)
    # convert pixel values (e.g. center)
    scaled_image = modellib.mold_image(image, cfg)
    # convert image into one sample
    sample = np.expand_dims(scaled_image, 0)
    # make prediction
    yhat = model.detect([image], verbose=0, mask_threshold=0.5)
    # extract results for first sample
    r = yhat[0]

    # compute gt_tot and pred_tot
    gt, pred = utils_confusion.gt_pred_lists(gt_class_id, gt_bbox, r['class_ids'], r['rois'])
    gt_tot = np.append(gt_tot, gt)
    pred_tot = np.append(pred_tot, pred)

    AP_, precision_, recall_, overlap_ = mrcnn.utils.compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"], r["class_ids"],
                                                        r["scores"], r['masks'])
    mAP_.append(AP_)

    tp, fp, fn = utils_confusion.plot_confusion_matrix_from_data(gt_tot, pred_tot, fz=18, figsize=(20, 20), lw=0.5)
    del tp[0]
    del fp[0]
    del fn[0]
    print("tp for each class :", tp)
    print("fp for each class :", fp)
    print("fn for each class :", fn)

confusion_matrix(val_set,model_inference,Icfg)

This gives the individual matrix+tp/fp/fn values for each image. How can I modify it so that I get the values/matrix of the combined dataset i.e. total number of fp,fn,tp.

Also is there a way to compute precision/recall from the entire dataset from this?

Hello
I copied your code to the end of my utils.py file.

I tried to calculate the mAP. I get this error:

AttributeError: module 'mrcnn.utils' has no attribute 'gt_pred_lists'

Your help will be appreciated.

Hi Altimis,

is there a way to implement True Negatives into the confusion matrix?

In my dataset, there are several images, that do not contain any object, that should be detected, and if the model correctly does not detect anything on that image, than I think that would count as a TN, right?

I would assume, this is only viable, if the TP,FP,TN,FN are considered imagewise and not detection wise, what do you think?

Thanks and best regards!

Hello Altimis,

I am using Mask-RCNN with 11 classes. I have been able to generate a confusion matrix using your repo.

The issue is that I'm only getting TPs. The FPs and FNs are always 0 for both validation and test datasets. However, the mAP for both is 0.85 and 0.918 respectively.

I've plotted the predictions as well for verification. The model mispredicts some of the classes and sometimes, it does not predict all of the classes in an image.

I fail to understand the issue here. Could you tell me what could be the problem?

Hey Altimis,

I used this repo to calculate the confusion matrix for classes (background and trees). I am really not sure about the totals in sum_lin and sum_col.

Here 56.79% is precision and 88.33% is recall for class trees right?

As the tp,fp, fn as per code are as follows:

As precision is calculated TP/(TP+FP) and Recall TP/(TP+FN).

Then I calculated mAP, mAR and F1 and got this

Shouldnt the mAP, mAR calculated and in confusion matrix be same or I am completely misunderstanding the orientation of false and trues in the matrix? Could you please help me understand it.

Many Thanks.

Hi, I don't understand really well where you setup the squares and nb of classes.
Is there some needs to change that into the utils.py function?

Hello @Altimis , I am getting a strange error when trying to print confusion matrix.Please help me in resolving this,

Could you please explain the illustration in the README a bit more? What do the green, what do the red values mean?

Hi there, I was able to plot the confusion matrix when I only have 1 class. However, when I have 2 different classes, I couldnt plot the confusion matrix using the code you given, could you tell me what was the problem? I have no idea what i should do to fix this issue....

the actual len of the pred vect is : 102
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 103
the actual len of the pred vect is : 103
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 104
the actual len of the pred vect is : 104
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 105
the actual len of the pred vect is : 105
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 106
the actual len of the pred vect is : 106
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 107
the actual len of the pred vect is : 107
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 108
the actual len of the pred vect is : 108
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 109
the actual len of the pred vect is : 109
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 110
the actual len of the pred vect is : 110
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 111
the actual len of the pred vect is : 111
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 112
the actual len of the pred vect is : 112
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 113
the actual len of the pred vect is : 113
The actual mean average precision for the whole images (matterport methode): 1.0

ValueError Traceback (most recent call last)

/usr/local/lib/python3.7/dist-packages/pandas/core/internals/managers.py in create_block_manager_from_blocks(blocks, axes)
1670 blocks = [
-> 1671 make_block(values=blocks[0], placement=slice(0, len(axes[0])))
1672 ]

6 frames

/usr/local/lib/python3.7/dist-packages/pandas/core/internals/blocks.py in make_block(values, placement, klass, ndim, dtype)
2743
-> 2744 return klass(values, ndim=ndim, placement=placement)
2745

/usr/local/lib/python3.7/dist-packages/pandas/core/internals/blocks.py in init(self, values, placement, ndim)
130 raise ValueError(
--> 131 f"Wrong number of items passed {len(self.values)}, "
132 f"placement implies {len(self.mgr_locs)}"

ValueError: Wrong number of items passed 2, placement implies 3

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last)

in ()
74 #print the confusion matrix and compute true postives, false positives and false negative for each class:
75 #ps : you can controle the figure size and text format by choosing the right values
---> 76 tp, fp, fn = utils.plot_confusion_matrix_from_data(gt_tot, pred_tot, dataset_test.class_names, fz=18, figsize=(20,20), lw=0.5)
77
78 print("mAP: ", np.mean(APs))

/usr/local/lib/python3.7/dist-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/utils.py in plot_confusion_matrix_from_data(y_test, predictions, columns, annot, cmap, fmt, fz, lw, cbar, figsize, show_null_values, pred_val_axis)
354
355 #plot
--> 356 df_cm = DataFrame(confm, index=columns, columns=columns)
357
358 pretty_plot_confusion_matrix(df_cm, fz=fz, cmap=cmap, figsize=figsize, show_null_values=show_null_values,

/usr/local/lib/python3.7/dist-packages/pandas/core/frame.py in init(self, data, index, columns, dtype, copy)
495 mgr = init_dict({data.name: data}, index, columns, dtype=dtype)
496 else:
--> 497 mgr = init_ndarray(data, index, columns, dtype=dtype, copy=copy)
498
499 # For data is list-like, or Iterable (will consume into list)

/usr/local/lib/python3.7/dist-packages/pandas/core/internals/construction.py in init_ndarray(values, index, columns, dtype, copy)
232 block_values = [values]
233
--> 234 return create_block_manager_from_blocks(block_values, [columns, index])
235
236

/usr/local/lib/python3.7/dist-packages/pandas/core/internals/managers.py in create_block_manager_from_blocks(blocks, axes)
1679 blocks = [getattr(b, "values", b) for b in blocks]
1680 tot_items = sum(b.shape[0] for b in blocks)
-> 1681 raise construction_error(tot_items, blocks[0].shape[1:], axes, e)
1682
1683

ValueError: Shape of passed values is (2, 2), indices imply (3, 3)

Hello Altimis,
My train data=476, Val data=95
Received : mAP =1, Precision=75% recall =63%
NUM_CLASSES = 1 + 2 (2 class and 1 background)

I referred your code for confusion matrix, I am confuse what is this confusion matrix is illustrating in percentage, can you please tell inferences of below confusion matix?

Thanks in advance

Hello @Altimis , I tried to use your code to generate confusion matrix , but I got the following error.
I have NUM_CLASSES = 1 + 4 (4 class and 1 background). please help me in resoving this issue .

Dear Altimis,
Many thanks for the code you provided it was really helpful, I got the confusion matrix and TP,FP and FN, but the precision recall curve does not show anything for me. do you know how can I solve this issue?

Hello Altimis,
You helped me a lot for calculating confusion matrix for Mask R-CNN. right now I trained YOLACT algorithm that is similar to Mask RCNN but I cannot create confusion matrix for that. Do you know how can I calculate it in the eval.py
the code link is here
https://github.com/dbolya/yolact/blob/master/eval.py
Thank you