cvmi-lab / slotcon Goto Github PK

Hi Xin Wen,

Thanks for your great work! Regarding SlotCon, I have two questions:
(1) I notice the prototypes are initialized with nn.Embedding. I am wondering how to ensure that the trainable prototypes are optimized to be meaningful semantic groups via backpropagation. Since the loss functions do not explicitly ensure this, I am a little bit confused about the optimization of prototypes.
(2) Have you tried how the resize operation in the data augmentation matters? I mean, if you only do crop along with other augmentation, without resize operation, will the performance drop heavily?

Thanks for your reply!

Hi, @xwen99 great work on unsupervised local features learning.

I am wondering how to evaluate SlotCon on unsupervised semantic segmentation benchmarks (i.e., Table 5 in the paper).
Do you have any plans to release the implementation?

When I run "python train_net.py --config-file /home/sri/SlotCon/transfer/detection/configs/COCO_R_50_FPN_1x_SlotCon.yaml --num-gpus 4 --resume MODEL.WEIGHTS /home/sri/SlotCon/sekhar.pkl OUTPUT_DIR /home/sri/SlotCon/output/COCO_R_50_FPN_1x_sekhar

it can not find the coco json file.

I am currently following the script for "Evaluation: Semantic Segmentation." However, when I try to follow the last "# run cityscapes" part, I encounter an error similar to the image below. I would like to know the solution to this issue.

KeyError: 'EncoderDecoder is not in the mmengine::model registry. Please check whether the value of EncoderDecoder

Can you provide the versions of these four packages?

mmcv
mmcv-full
mmengine
mmsegmentation

Hello,

Thank you for your very interesting work ! I'm currently trying to replicate your results with your provided codebase and I was wondering whether you also tested a Vision Transformer architecture as encoder ? You compared in the paper with DINO, but I wanted to know if you where able to get some properties close to what they obtained (a kind of saliency map with the attention map around the object of interest).

Thank you again for your response !

Hi, Xin wen!

I am trying to use SlotCon to process 3D data, but some problems hinder me. Such as the operation torchvision.ops.roi_align don't provide 3D version. Do you have some solutions for that?

Hi Xin,

Thanks for the great and insightful work.

When I read the code, I am confused by the label generation for contrastive learning of slots.
As shown in https://github.com/CVMI-Lab/SlotCon/blob/main/models/slotcon.py#L186, the slots with the same indexes are viewed as positive indexes while I find that these slots are generated by masked pooling from features and indexes maybe not be related to the semantic classes. Maybe I have missed something.

Looking forward to your rely!

Respected Sir,

While I am executing the code, getting the following error. It will be a very helpful for me if you are getting me out of that error.

Traceback (most recent call last):
File "/transfer/detection/convert_pretrain_to_d2.py", line 19, in
input = sys.argv[1]
IndexError: list index out of range