License: Other

MATLAB 100.00%

visual-relationship-detection's Introduction

Visual Relationship Detection with Language Priors

Cewu Lu*, Ranjay Krishna*, Michael Bernstein, Li Fei-Fei, European Conference on Computer Vision (ECCV), 2016, (oral). (* = indicates equal contribution)

Citation

@InProceedings{lu2016visual,
   title = {Visual Relationship Detection with Language Priors},
   author = {Lu, Cewu and Krishna, Ranjay and Bernstein, Michael and Fei-Fei, Li},
   booktitle = {European Conference on Computer Vision},
   year = {2016},
 }

Introduction

Visual Relationship Detection is a task where the input is a simple image and the model is supposed to predict a set of relationships that are true about the image. There relationships are in the form of . Visual Relationship Detection is a natural extension to object detection where we now not only detect the objects in an image but predict how they are interacting with one another.

Detailed description of the task and our model is provided in our paper at ECCV 2016.

##Licence

This software is being made available for research purpose only. Check LICENSE file for details.

System Requirements

This software is tested on Ubuntu 14.04 LTS (64bit).

MATLAB (tested with 2014b on 64-bit Linux) Caffe

Demo

To view some example results, you can directly run demo.m on about 50 images. The results will be saved in "results/demo" and "results/demo_zeroShot".

>> run demo.m;

Relationship and Preidcate Detection

To detect relationship in our dataset, please run relationship_phrase_detection.m. Each image will have a set of relationships (<subject, predicate, object>) predicted with bounding boxes for the subjects and objects as bounding boxes. The relationship detection results (including zero-shot preformance) will also be reported. Predicate detection results can be obtained by running predicate_detection.m.

For each image, we provide our VGG based object detection results (object category, confidence score and bounding boxes) and CNN scores on the union of the bounding boxes of pairs of participating objects in each relationship. You can also train your own object detection models using newer state of the art architechures like ResNet.

>> run relationship_phrase_detection.m;
>> run predicate_detection.m;

visual-relationship-detection's People

Contributors

Stargazers

Watchers

visual-relationship-detection's Issues

Training code and pre-trained model?

Hi,

Thanks for the code. It is a great work!

But can you also provide the training code? Then we can train our own W and b in Eq. (2).
And is that possible to provide your pre-trained model for the object detection and predicates prediction model, so we can extract the 'objectDetRCNN.mat' and 'UnionCNNfeaPredicate.mat' for our own images.

Thanks!

How to train Union CNN Score?

Hello,How train Union CNN Score? Can you release the training code?

How to build my own "VRD"

Hello，your paper help me a lot and I want to build my own dataset. But I don't know how to generate the json file conveniently. Do you have any tools to generate them(like labelImg).
look forward to your reply.

The Evaluation of Predicate Detection

Hello,

I'm confused about the evaluation of predicate detection.
The paper says

our input is an image and set of localized objects. The task is to predict a set of possible predicates between pairs of objects

But in predicate_detection.m, the predictions are made for each groundtruth relation. It takes a groundtruth (subject, object) pair as input and classifies the predicate.
If that is the case, the model has access to not only an image and set of localized objects, but also:

Whether or not a pair of objects have a relation
(If they do have certain relation) The order of the two objects (which one is the subject and which one is the object)

Could you please clarify what is the correct setup for the predicate detection task? Thanks.

Data for Zero-shot Learning and Image Retrieval

Hi,

Can you share with us the data for evaluating the zero-shot learning and image retrieval as stated in the paper?

Best,

How can i download the dataset in the paper?

Where can I find the images ?

I am unable to figure out where to download the images. Thank You in advance

请问有tensorflow版本的吗？

Misusing MATLAB function "length"

Hi,

Thanks you for the effort of releasing the source code.
In top_recall_Relationship.m, when calculating num_pos_tuple, you used
length(gt_tuple_label{ii}). length gives the dimension of the longer axis, which results in an erroneous denominator when calculating recall score. Would you consider fix and use size(gt_tuple_label{ii}, 1) instead?

prof-lu-cewu / visual-relationship-detection Goto Github PK