vibashan / irg-sfda Goto Github PK

The test result of the teacher model is:

The test result of the student model is:
![image](https://github.com/user-attachments/assets/edd89032-8657-4820-a7f4-13938ba3e3b9
The test results of the teacher model are similar to those presented in the paper . But the final model saved by the code is the student model. Which model is used for testing, the teacher model or the student model?

Thanks for your contributions! I find all of the configs with the IMS_PER_BATCH=1, and

The features outputs by ROI_pool and the matrix multiple operation indicate that only single batch is supported during training?

Hello, I am reproducing your article. Can you provide the entire structure of the dataset? After downloading all the datasets, I couldn't find the correct structure of the dataset. Thank you.

Hello,

Can you provide the instructions/code you used to train the source models?

Thank you for your work

Hello, I encountered this error while configuring the environment. How can I resolve it? python -m pip install -e irg-sfda-main

Hii,

=I am getting loss_bbox_reg - 0 for all, anything can help me in this regard. Thanks

Hi, thanks for your release.
I have read your code and paper. In your paper, the positive sample mask is obtained from the feature after GCN. But in your code, you maybe use the feature before GCN(s_box_features) to obtain the positive sample mask. How to understand

    s_box_features = self.roi_heads._shared_roi_transform([features['res4']], [t_proposals[0].proposal_boxes]) #t_proposals[0], results[1]
    s_roih_logits = self.roi_heads.box_predictor(s_box_features.mean(dim=[2, 3]))

    t_box_features = model_teacher.roi_heads._shared_roi_transform([t_features['res4']], [t_proposals[0].proposal_boxes])
    t_roih_logits = model_teacher.roi_heads.box_predictor(t_box_features.mean(dim=[2, 3]))

    s_graph_feat = self.GraphCN(s_box_features.mean(dim=[2, 3]))
    s_graph_logits = self.roi_heads.box_predictor(s_graph_feat)
    
    t_graph_feat = self.GraphCN(t_box_features.mean(dim=[2, 3]))
    t_graph_logits = model_teacher.roi_heads.box_predictor(t_graph_feat)

    losses["st_const"] = self.KD_loss(s_roih_logits[0], t_roih_logits[0]) 
    losses["s_graph_const"] = self.KD_loss(s_graph_logits[0], s_roih_logits[0]) 
    losses["t_graph_const"] = self.KD_loss(t_graph_logits[0], t_roih_logits[0]) 
    losses["graph_conloss"] = self.Graph_conloss(t_box_features.mean(dim=[2, 3]), s_box_features.mean(dim=[2, 3]), self.GraphCN)
    #s_box_features 

#Graph_conloss
def forward(self, t_feat, s_feat, graph_cn, labels=None, mask=None):    

        qx = graph_cn.graph.wq(s_feat)
        kx = graph_cn.graph.wk(s_feat)        
        sim_mat = qx.matmul(kx.transpose(-1, -2))
        dot_mat = sim_mat.detach().clone()

        thresh = 0.5
        dot_mat -= dot_mat.min(1, keepdim=True)[0]
        dot_mat /= dot_mat.max(1, keepdim=True)[0]
        mask = ((dot_mat>thresh)*1).detach().clone()
        mask.fill_diagonal_(1)

        anchor_feat = self.head_1(s_feat)
        contrast_feat = self.head_2(s_feat)

        anchor_feat = F.normalize(anchor_feat, dim=1)
        contrast_feat = F.normalize(contrast_feat, dim=1)

        ss_anchor_dot_contrast = torch.div(torch.matmul(anchor_feat, contrast_feat.T), self.temperature)  ##### torch.Size([6, 6])
        logits_max, _ = torch.max(ss_anchor_dot_contrast, dim=1, keepdim=True)  ##### torch.Size([6, 1]) - contains max value along dim=1
        ss_graph_logits = ss_anchor_dot_contrast - logits_max.detach()

        ss_graph_all_logits = torch.exp(ss_graph_logits)
        ss_log_prob = ss_graph_logits - torch.log(ss_graph_all_logits.sum(1, keepdim=True))
        ss_mean_log_prob_pos = (mask * ss_log_prob).sum(1) / mask.sum(1)  
    
        # loss
        ss_loss = - (self.temperature / self.base_temperature) * ss_mean_log_prob_pos
        ss_loss = ss_loss.mean()

        return ss_loss

Why the example evaluation scripts

CUDA_VISIBLE_DEVICES=$GPU_ID python tools/plain_test_net.py --eval-only \
--config-file configs/sfda/foggy_baseline.yaml --model-dir $PATH TO CHECKPOINT

regarding foggy could be runnable when I don't make any foggy_cityscape dataset?

Also, i CANT find the relevant info about adding fog in FoggyDetectionEvaluator. i AM so confused.

Hi, when I run the comand 'python tools/train_st_sfda_net.py --config-file configs/sfda/sfda_foggy.yaml', it has an error like this:

And I cannot find any "ttda_rcnn" file, could you help me? Thanks a lot.

Dear Dr. Vibashan,

I am a graduate student from Changchun University of Science and Technology in China, focusing on machine learning and saliency prediction.

Recently, I have had the opportunity to read your paper, "ST-MTL: Spatio-Temporal Multitask Learning Model to Predict Scanpath While Tracking Instruments in Robotic Surgery". I find the research intriguing and have been studying it in detail. I am particularly interested in the prediction of saliency maps in surgical scenarios.

However, the training sets 1, 2, 3, 5, 6, and 8 are not publicly available. Additionally, the dataset from the 2017 MICCAI Surgical Instrument Segmentation Challenge only contains ground truth segmentation masks and lacks ground truth saliency maps.

I would be immensely grateful if you could share the complete dataset and any relevant information that may be essential for my research. I assure you that the dataset will be utilized solely for academic purposes.

I am deeply appreciative of your consideration and look forward to your response at your earliest convenience.

Sincerely,
Yuqi Yan

Hi,

Thank you for this codebase. Please could I just check where the augmentations of the input image happen in the code?

Thank you,
Best wishes,
Emily

TEST:
AUG:
ENABLED: false
FLIP: true
MAX_SIZE: 4000
MIN_SIZES:
- 400
- 500
- 600
- 700
- 800
- 900
- 1000
- 1100
- 1200
DETECTIONS_PER_IMAGE: 100
EVAL_PERIOD: 0
EXPECTED_RESULTS: []
KEYPOINT_OKS_SIGMAS: []
PRECISE_BN:
ENABLED: false
NUM_ITER: 200
VERSION: 2
VIS_PERIOD: 0

[12/29 16:43:20 detectron2]: Full config saved to ./checkpoint/clipart/config.yaml
[12/29 16:43:20 d2.utils.env]: Using a generated random seed 20643623

I had to wait half an hour to load the program, which I ran with the RTX 3090.
Do you know how I can solve this problem?
Thank you very much!

I know how to regiter custom dataset, but how to train source model on custom dataset?

Hi! I have some problems when I want to reproduce the results.
When I want to change the path of dataset, I get the error:
AssertionError: Dataset 'cityscape_2007_train_s' is already registered!
Can you provide some suggestions for me? Thank you!

Thanks for your great work.

I am trying to reproduce the results, but get stuck in installing detectron2 environments.

When I follow the Detectron2 installation , install detectron2 via following command:

python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'

I have installed detectron2 successfully. But when I train the model, I get the error:

ImportError: libcudart.so.11.0: cannot open shared object file: No such file or directory

I found the project has directory detectron2, thus we need to install detecron2 by:

python -m pip install -e detectron2

But I get another error:

ERROR: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode: /mnt/data0/home/codebase/irg-sfda/detectron2

Can the author provide some suggestions for me?

Update:

I install detectron2 via:

python -m pip install .

And then run the train script and raise an error:

Hi Mr. Vibashan,

I would like to reproduce your paper but I only get AP values of 0 when I test with your uploaded model with this code:

python tools/plain_test_net.py --eval-only --config-file configs/sfda/foggy_baseline.yaml
--model-dir model/model_teacher_10.pth

I get the following output:
[08/10 15:36:07 fvcore.common.checkpoint]: [Checkpointer] Loading from model/model_teacher_10.pth ...
WARNING [08/10 15:36:07 fvcore.common.checkpoint]: Some model parameters or buffers are not found in the checkpoint:
GraphCN.gc1.{bias, weight}
GraphCN.gc2.{bias, weight}
GraphCN.gc3.{bias, weight}
GraphCN.graph.wk.{bias, weight}
GraphCN.graph.wq.{bias, weight}
Graph_conloss.head_1.0.{bias, weight}
Graph_conloss.head_1.2.{bias, weight}
Graph_conloss.head_2.0.{bias, weight}
Graph_conloss.head_2.2.{bias, weight}
[08/10 15:36:07 detectron2]: Trained model has been sucessfully loaded
...
[08/10 15:37:25 detectron2]: Evaluation results for cityscape_2007_test_t in csv format:
[08/10 15:37:25 d2.evaluation.testing]: copypaste: Task: bbox
[08/10 15:37:25 d2.evaluation.testing]: copypaste: AP,AP50,AP75
[08/10 15:37:25 d2.evaluation.testing]: copypaste: 0.0000,0.0002,0.0000

I also get an AP of 0 after the training with foggy cityscapes with the base source model. I have not altered the code. Do you have any idea what the problem could be?

Thank you for your work and kind regards,
Anna

I noted that the weights of the teacher network are updated every epoch? Usually, we update the teacher model every iteration. Why do authors choose such a strategy?

Also, the paper states the =keep_rate= for teacher update is set to 0.99, and the code here is set to 0.9.