This repository is a PyTorch implementation for generalized few-shot semantic segmentation. It is developed based on this repo for semantic segmentation / scene parsing.

1. Requirement:

   • Hardware: better with >=11G GPU memory
   • Software: PyTorch>=1.1.0, Python3, tensorboardX,

   env.yml is an environment file that works on Dart machine. You can clone the environment by running:

   conda env create -f env.yml

2. Clone the repository:

   git clone [email protected]:Yinan-Zhao/gfs_seg.git

3. Dataset and Initialization:

   • Download PASCAL and COCO datasets from here and put them under folder datasets(you can alternatively modify the relevant paths specified in folder config). COCO and pascal should be found under datasets.

     cd semseg
     mkdir -p datasets
     

   • Download ImageNet pre-trained models and put them under folder initmodel for weight initialization. Remember to use the right dataset format detailed in FAQ.md.

4. The First Training Stage: Base Training

   • Specify the gpu id used in config and the fold id, then do training:

     sh tool/train_normal.sh pascal pspnet50_normal_split0

     sh tool/train_normal.sh coco pspnet50_normal_split0

     If split id is specified as 100, it will train with all the training images.

5. The Second Training Stage: Few-Shot Fine-tuning

   • Specify weight as the trained model we obtain in the first base training stage and # of shots in config, then do fine-tuning:

     sh tool/train_finetune.sh pascal pspnet50_normal_finetune_split0_shot10_lr1e2

     sh tool/train_finetune.sh coco pspnet50_normal_finetune_split0_shot10_lr1e2

6. Test:

   • Download trained segmentation models and put them under folder specified in config or modify the specified paths.

   • For full testing (get listed performance):

     sh tool/test_normal.sh coco pspnet50_normal_finetune_split0_shot5_lr1e2

7. Visualization: tensorboardX incorporated for better visualization.

   tensorboard --logdir=exp/voc2012/pspnet50_normal_finetune_split0_shot5_lr1e2/

• Trained Models
• Method Figure
• Few-shot object detection (paper), Few-shot image classification (paper1), (paper2)
• Data analysis tool: t-SNE
• Datasets: attributes (names and colors) are in folder data and some sample lists can be accessed.
• Some FAQs from the original repo: FAQ.md.

• Similarity and difference between different few-shot tasks (semantic segmentation, object detection and image classification). Although fine-tuning based approaches have shown success in each task, the set of parameters that need to fine-tuned are different (for segmentation we fine-tune all the layers after the ResNet backbone while for object detection only the last conv layer is fine-tuned). The different set of parameters that need to be fine-tuned indicates the difference in terms of the generalization of the learned features.

• For any questions, please contact Yinan Zhao ([email protected])