Giter VIP home page Giter VIP logo

zero_shot_detection's Introduction

PWC

PWC

PWC

PWC

Code for ACCV 2020 Synthesizing the Unseen for Zero-shot Object Detection

Zero Shot Detection (ZSD) is a recently introduced paradigm which enables simultaneous localization and classification of previously unseen objects. It is arguably the most extreme case of learning with minimal supervision. we propose a symantically driven conditional feature generation module to synthesize visual features for unseen objects.

Feature Generation Pipeline

Feature Synthesizer

t-SNE Visualization

Set up the environment

  • mmdetection we recommend using Docker 2.0. Please use the mmdetection codes from this repo.
  • conda env create -f environment.yml

The following scripts are for dfferents steps in the pipeline on MSCOCO dataset, please see the respective files for more arguments. Before running the scripts, please set the datasets and backbone paths in the config files. Weights of ResNet101 trained excluding overlapping unseen classes from ImageNet. Weights of FasterRCNN trained on MSCOCO2014, PASCALVOC, ILSVRC

1. Train Detector

cd mmdetection
./tools/dist_train.sh configs/faster_rcnn_r101_fpn_1x.py 8 --validate

2. extract features

cd mmdetection
# extract seen classes features to train Synthesizer and unseen class features for cross validation
python tools/zero_shot_utils.py configs/faster_rcnn_r101_fpn_1x.py --classes [seen, unseen] --load_from [detector checkpoint path] --save_dir [path to save features] --data_split [train, test]

# example to extract training features for seen classes

python tools/zero_shot_utils.py configs/faster_rcnn_r101_fpn_1x.py --classes seen --load_from ./work_dir/coco2014/epoch_12.pth --save_dir ../../data/coco --data_split train

# example to extract test features for unseen classes

python tools/zero_shot_utils.py configs/faster_rcnn_r101_fpn_1x.py --classes unseen --load_from ./work_dir/coco2014/epoch_12.pth --save_dir ../../data/coco --data_split test

3. Train Generator

# modify the paths to extracted features, labels and model checkpoints. 
./script/train_coco_generator_65_15.sh

4. Evaluate

cd mmdetection
    ./tools/dist_test.sh configs/faster_rcnn_r101_fpn_1x.py [detector checkpoint path for seen classes] [num of gpus] --dataset [coco, voc, imagenet] --out [file name to save detection results] [--zsd, --gzsd] --syn_weights [path to synthesized classifier checkpoint]

# example 

./tools/dist_test.sh configs/faster_rcnn_r101_fpn_1x.py work_dir/coco2014/epoch_12.pth 8 --dataset coco --out coco_results.pkl --zsd --syn_weights ../checkpoints/coco_65_15/classifier_best_137.pth

Citation

If you use this code for your research, please consider citing:

    @misc{hayat2020synthesizing,
      title={Synthesizing the Unseen for Zero-shot Object Detection}, 
      author={Nasir Hayat and Munawar Hayat and Shafin Rahman and Salman Khan and Syed Waqas Zamir and Fahad Shahbaz Khan},
      year={2020},
      eprint={2010.09425},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
    }

Results

  • MSCOCO

  • MSCOCO Class Wise AP

  • ILSVRC

  • PASCAL VOC

Qualitative Results

  • Zero Shot Detections

  • Generalized Zero Shot Detections

zero_shot_detection's People

Contributors

cmbb2000 avatar nasir6 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

zero_shot_detection's Issues

Extract training features

I am following your code in extracting features to create the t-SNE.
In the feats_extract function, what is the use of sub_neg_inds at line #324

mmdetect version

Hello @nasir6

Hope you are doing great.
It would be really helpful if you could share the mmdet and mmcv version used for running mmdetection.

Best,
Akshita

Training Time

Nice work.

How long does your model need to train?

Unseen classifier loss throws error

Hello @nasir,

The unseen classifier used while training GAN throws a input data range error. The input I am getting for the loss is between -1 to 1 and it expects data netween 0 to 1.
It would really helpful if you have a look at the issue.

Best,
Akshita

Missing attributes

Hi @nasir6,

It would be really helpful if you could share the missing attributes file, fasttext.npy for MS-COCO.

Best,
Akshita

Can't find voc_fc_1024_300.pth

When I run 'train_voc_generator.sh',can't find the 'voc_fc_1024_300.pth' pretrained file, could you please upload it. Thanks

True number of training samples for the training set in VOC 2007

In fact, I looked meanwhile other literature and most of the literature wrote using 2072 images from voc 2007, but in the code you provided, I could only get 1622 images that could be used for training (after excluding images that appear to be unseen classes).

I'm wondering if there are any that I need to be aware of additionally. Looking forward to your reply. Thanks.

How to get the Qualitative Results.

Hi, thank you for providing the excellent repository!
After reproducing your results, I want to get the detection boxes for zero-shot detection or generalized zero-shot detection on the images.
That is how to get your Qualitative Results.
Look forward to your reply.

[Errno 2] 没有那个文件或目录: 'VOC/fasttext_multilabels_4.npy'

Excuse me, I can't find this file, what should I do?

Traceback (most recent call last):
File "/home/luyue/.local/lib/python3.6/site-packages/numpy/lib/npyio.py", line 416, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] 没有那个文件或目录: 'VOC/fasttext_multilabels_4.npy'

When I tested with gzsd, an error was reported

/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [23,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [26,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [29,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [32,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [35,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [38,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [41,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [44,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [47,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [50,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [53,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [56,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [59,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [62,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [65,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [68,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [71,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [74,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [77,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [80,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [83,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [86,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [89,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [92,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [2,0,0], thread: [95,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
Traceback (most recent call last):
File "tools/test.py", line 263, in
main()
File "tools/test.py", line 245, in main
outputs = single_gpu_test(model, data_loader, args.show)
File "tools/test.py", line 27, in single_gpu_test
result = model(return_loss=False, rescale=not show, **data)
File "/root/miniconda3/envs/zsd/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/miniconda3/envs/zsd/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "/root/miniconda3/envs/zsd/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/autodl-tmp/RRFS/RRFS-main-NWPU/mmdetection/mmdet/core/fp16/decorators.py", line 49, in new_func
return old_func(*args, **kwargs)
File "/root/autodl-tmp/RRFS/RRFS-main-NWPU/mmdetection/mmdet/models/detectors/faster_rcnn.py", line 73, in forward
return self.forward_test(img, img_meta, **kwargs)
File "/root/autodl-tmp/RRFS/RRFS-main-NWPU/mmdetection/mmdet/models/detectors/base.py", line 102, in forward_test
return self.simple_test(imgs[0], img_metas[0], **kwargs)
File "/root/autodl-tmp/RRFS/RRFS-main-NWPU/mmdetection/mmdet/models/detectors/two_stage.py", line 274, in simple_test
x, img_meta, proposal_list, self.test_cfg.rcnn, rescale=rescale)
File "/root/autodl-tmp/RRFS/RRFS-main-NWPU/mmdetection/mmdet/models/detectors/test_mixins.py", line 120, in simple_test_bboxes
cls_score, bbox_pred = self.test_gzsd(rcnn_test_cfg, cls_score, bbox_pred)
File "/root/autodl-tmp/RRFS/RRFS-main-NWPU/mmdetection/mmdet/models/detectors/test_mixins.py", line 73, in test_gzsd
cls_score[:, seen_class_inds_bg] = F.softmax(cls_score[:, seen_class_inds_bg], dim=1)
RuntimeError: CUDA error: device-side assert triggered
Aborted (core dumped)

When I used gzsd to test, I reported the above error, how to solve it ??

Generate vocabulary file for Pascal VOC dataset

Hi,
Thank you for providing the awesome repository!
We tried to change the dataset from COCO to Pascal VOC 2007; however, the there only contains attribute file.

Could you share the method to generate the vocabulary file for Pascal VOC 2007?
Thank you

Reproducibilty problem

Hello @nasir,

I am not able to reproduce the numbers mentioned in the paper. I am getiing a classification accuracy of 46.7. It would be really helpful if you could have a look at the issue.

Inference to single image

Hi, Thank you for your work.

I would like to inquire about the possibility of adding inference code for a custom single image.
I am interested in conducting a qualitative evaluation of your work.

How to generate class embedding files.

Hi, thanks for you great job.
I just get confused how do you generate these classes embedding files(fastext, glove).
How does the index in classes embedding files match to the class id?
Could provide a little more details about generating class embedding files?
Thanks!

Running about the DDP problem

Thanks for your excellent works,
i can successfully run the code, but when i train the detector with 8 gpus, the forth gpu is always out of memory, where the others have many memories.
Look forward to your reply

How to get VOC results?

Hi,

Nice work!
I was very sorry to bother you.
When I used your code, I found I could not get VOC result, where the instructions of voc?

Thanks very much!

mmdet and mmcv version issue

Hello @nasir6

Hope you are good.
It would be really helpful if you could share the mmdet and mmcv version used for running mmdetection.
thanks,
Sushil Kumar

How to compute Recall@100

Hi, one of the evaluation metric for zero shot object detection is Recall@100, but how to compute it is not very clear.
My understanding of computation process is following.
First, select top 100 detections from an image.
Second, mark a predicted bounding box as positive if it has an IoU greater than a threshold (0.5 for example) and no other higher confidence bounding box has been assigned to the same GT box.
Third, compute recall@100 for this image number_of_positive_prediction / 100.
Forth, compute recall@100 for all images sum(recall@100 for each image) / number_of_image.

Is it correct ? Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.