Question about recall @ k about vsepp HOT 3 CLOSED

The definition of precision and recall here are closer to what is used in NLP and in defining metrics such as ROUGE_L and ROUGE_S. In that context, a model is returning a sequence of tokens (words) and it is evaluated based on how similar that sequence is to one or multiple ground-truth sequences. If there are multiple ground-truth sequences, the score is the maximum of the score with any of the ground-truths. In the case of caption retrieval, it is common to use this in-between precision-recall metric that's treating full-length sequences as single tokens. That is just due to the simplification made to the task where instead of generating the captions, we use a database of ready-to-use captions. But it still makes more sense to use the ROUGE version of the precision-recall metrics rather than the document retrieval version.

Lin 2004, "ROUGE: A Package for Automatic Evaluation of Summaries"
Chen 2015, "Microsoft COCO Captions: Data Collection and Evaluation Server"

from vsepp.

Yeah. I agree that it's confusing.

from vsepp.

Thanks for the reply! I see what you mean, and -- looking back at even early retrieval reporting on MSCOCO (e.g., Karpathy and Li 2015) -- the score is clearly not capped at .2. I agree that this metric makes sense, though it really shouldn't be called recall IMO. Perhaps "hit-rate-at-k" is better? Anyway -- this seems to be a problem that I have with nomenclature more than anything else. Thanks again for the code!!! :)

from vsepp.