Sampling output image visualization problem? about medsegdiff HOT 9 CLOSED

@WangPing521 If I understand correctly, the first step in "run_step" concatenates the image and mask together, and leave "cond" as an empty dictionary. For ISIC dataset, I think the image size is [B, 3, H, W] and the mask is [B, 1, H, W].

from medsegdiff.

is this the ensemble result saved in your result file? Some others met the similar problem before #6.

from medsegdiff.

Yes, using the savedmodel030000.pt and these images are named ''{slice_id}_output_ens.jpg'

from medsegdiff.

This is one of the predicted masks after I trained the model on ISIC dataset

from medsegdiff.

it's close. Note that MedSegDiff will outcome uncertianty-aware segmentation, because of the uncertianty nature of medical image segmentation(check: Learning calibrated medical image segmentation via multi-rater agreement modeling(CVPR 2021), Learning self-calibrated optic disc and cup segmentation from multi-rater annotations(MICCAI 2022)). but your prediction seems a little too much, you may want to generate more confident results by more training or ensemble.

if you want to obtain the binary results, you can threshold and postprocess this prediction. I used the postprocess config in nnUNet for example.

from medsegdiff.

Thanks for confirming,

Does the number of channels (RGB vs Grayscale) in the input images matter? Or should all grayscale input images be converted into RGB first?

The custom dataset that i have are grayscale images. I am converting it to RGB first, and using args.in_ch = 4 for training.

from medsegdiff.

it's close. Note that MedSegDiff will outcome uncertianty-aware segmentation, because of the uncertianty nature of medical image segmentation(check: Learning calibrated medical image segmentation via multi-rater agreement modeling(CVPR 2021), Learning self-calibrated optic disc and cup segmentation from multi-rater annotations(MICCAI 2022)). but your prediction seems a little too much, you may want to generate more confident results by more training or ensemble.

if you want to obtain the binary results, you can threshold and postprocess this prediction. I used the postprocess config in nnUNet for example.

I want to know what is the input of diffusion model, in order to get a segmentation? For example, we have a 5-class within an image. Then we have two choice, one is to use the one-hot (1, 5, h, w) where the values are 0 and 1, suppose batch_size is 1, as the input of diffusion model; another is to use the mask(1,1,h,w) where the values are 0, 1, 2, 3, 4, as the input of diffusion model. Which one is you choice? In this diffusion on segmentation rather than image case, what are the output during the reverse stage? How to map the sampled output to the labels(0 and1, or 0,1,2,3,4)?

If i am wrong above, as you said "MedSegDiff will outcome uncertianty-aware segmentation", the output is probability map like the output of normal segmentation network? Can you give me some hints about this? I am very confusing about the diffusion on segmentation rather than image.

I would very appreciate if you can help.

from medsegdiff.

@WangPing521 It depends on your dataset, both of them can work. if it was me, i will use (1, 5, h, w). diffusion model outputs a residual noise. Then substract it from the last noisy result(segmentation) to get a clearer one. The iteration enables it to generate a segmentation from a gaussian noise conditioned by the raw image. you can refer to our paper for more details.

from medsegdiff.

Thanks for your reply!

I have a naive question for your input of code, "self.run_step(batch, cond) ", here, Does the batch mean the label, the cond mean the original image? Can you show the dimensionality of these two tensors in a (N, C, H, W) way?

from medsegdiff.