This repository contains the official source code for our paper:
Shatter and Gather: Learning Referring Image Segmentation with Text Supervision
Dongwon Kim1, Namyup Kim1, Cuiling Lan2, and Suha Kwak1
1POSTECH CSE, 2Microsoft Research Asia
ICCV, Paris, 2023.
- Download the MS COCO images are under
data/coco/images/train2014/ - Download the ReferItGame data are under
data/referit/imagesanddata/referit/mask - Download TF-resnet and TF-deeplab under
externalfolder. Then strictly foll - Download refer under
external. Then strictly follow the Setup and Download section. Also put thereferfolder inPYTHONPATH - Download the MS COCO API also under
external(i.e.external/coco/PythonAPI/pycocotools)
python build_batches.py -d Gref -t train
python build_batches.py -d Gref -t val
python build_batches.py -d unc -t train
python build_batches.py -d unc -t val
python build_batches.py -d unc -t testA
python build_batches.py -d unc -t testB
python build_batches.py -d unc+ -t train
python build_batches.py -d unc+ -t val
python build_batches.py -d unc+ -t testA
python build_batches.py -d unc+ -t testB
./data
├─ refcoco
│ ├─ Gref
│ │ ├─ train_batch
│ │ │ ├─ Gref_train_0.npz
│ │ │ ├─ Gref_train_1.npz
│ │ │ └─ ...
│ │ ├─ train_image
│ │ ├─ train_label
│ │ ├─ val_batch
│ │ ├─ val_image
│ │ └─ val_label
│ ├─ unc
│ │ └─ ...
│ └─ unc+
│ └─ ...
├─ phrasecut
│ └─ images
│ ├─ refer_train_ris.json
│ ├─ refer_val_ris.json
│ └─ refer_test_ris.json
├─ Gref_emb.npy
├─ referit_emb.npy
├─ vocabulary_Gref.txt
└─ vocabulary_referit.txt
- Python 3.10.9
- PyTorch 1.13.1+cu117
Instructions:
conda create -n sag python=3.10 -y
conda activate sag
pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu11
pip install einops tqdm wandb transformers
pip install matplotlib timm opencv-pythonsh ./train_eval_gref.sh # Gref
sh ./train_eval_unc.sh # UNC
sh ./train_eval_unc+.sh # UNC+Parts of our codes are adopted from the following repositories.
Dataset Setup instruction is from TF-phrasecut-public repository.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent