Query regarding Pretrained model about vl-bert HOT 3 CLOSED

1. Actually there are only two tasks in pre-training: Masked Language Modeling with Visual Clues, Masked RoI Classification with Linguistic Clues. Sure, the pre-trained models provided in this repo have been pre-trained on these tasks, you don't need to perform pre-training by yourself.
2. We use Conceptual Captions for both tasks, and English Wikipedia & BookCorpus only for Masked Language Modeling task.
3. Could you provide the detailed link to the line of code since I am not sure where do you exactly mean?

Thanks!

from vl-bert.

Thanks for the reply.

3. For (3), I meant this line of code. Is it only partially loading the weights ?

I have some more questions.

4. Could you also share the weights for fast rcnn module ? It seems that the pretrained model is meant for ResNetVLBert only.
5. Do I need to train fast rcnn on Visual Genome for my task?
6. I want to port fast rcnn to torchvision's faster rcnn. What does obj_reps actually represent here, are these the box predictions coming out of ROI Head ? And you seem to be using 'cnn_regularization' loss which is cross entropy loss for classes, what about MSE loss for regression of bounding boxes ? I think MSE loss would be necessary for improving region proposals.
7. If I were to port to torchvision's faster_rcnn implementation, please see here, Will detections from self.transform.postprocess be 'obj_reps' and cross entropy loss be 'cnn_regularization' loss ?

from vl-bert.

@prajjwal1
3. The partial loading is because downstream task and pre-training task may have different prediction heads, and only common weights (including the Fast RCNN, VL-BERT, and maybe some common heads) are loaded.
4. Actually, the weights of Fast RCNN is included:

• (Offline) Use the pre-trained Faster RCNN to extract bounding boxes and store them (refer to

  VL-BERT/data/conceptual-captions/ReadMe.txt

  Line 59 in 28cbde9

  9. python ./tools/generate_tsv_v2.py --gpu 0,1,2,3,4,5,6,7 --cfg experiments/cfgs/faster_rcnn_end2end_resnet.yml --def models/vg/ResNet-101/faster_rcnn_end2end_final/test.prototxt --net data/faster_rcnn_models/resnet101_faster_rcnn_final.caffemodel --split conceptual_captions_train --data_root {Conceptual_Captions_Root} --out {Conceptual_Captions_Root}/train_frcnn/

  );
• Use only the backbone (ResNet) and RoI head of pre-trained Faster RCNN to initialize our Fast RCNN;
• (Online) Treat precomputed boxes as region proposals, and using Fast RCNN to extract their features.

P.S.
The obj_reps refers to visual feature of each RoI used in VL-BERT, obj_reps_raw means RoI features coming out of RoI head, and the cnn_regularization is deprecated, we don't use it in both pre-training and fine-tuning.

from vl-bert.