Enquiries about the dataset about ucl-affcorrs HOT 1 CLOSED

This comment contains my email response to your questions from November. I am attaching here for future reference.

Dear JIXIN,
Here's my answers to your enquiries:

1.The 'ground truth' is the mask provided for each image - e.g. If using item A with mask A as input, and querying on item B - the expected outcome is mask B. Note the affordances masks are stored in the numpy file - see Ans. 2. Below.

2. The seven channels represent the 7 affordances, this is taken as in the original full UMD dataset, as far as I remember. For some objects you will find them empty, e.g. For bowl there is no scoop.

3. Instructions on how evaluations are done are provided in the README of the dataset folder. At this time we don't provide the code for this but they are quite simple to write and implement if you follow the pseudocode provided.

'Each pixel is labeled with rankings for the 7 different affordances,
where the integer at gt_label(i,j,k) indicates the rank of the kth
affordance. A value of 1 is the highest ranked (most likely) label,
and we allow for ties in the ranking. Background pixels are not labeled,
and have all zero values.

The ground truth labels are:

1 - 'grasp'
2 - 'cut'
3 - 'scoop'
4 - 'contain'
5 - 'pound'
6 - 'support'
7 - 'wrap-grasp'

As for the sizes, I guess resizing is needed, which is also why researchers use the Fw score metric since it accounts for ground truth mask resizing. Note the data you see in UMDi dataset is all a subpart of the original UMD dataset.

For each class - you evaluate the other classes (for inter-class transfer), the rank of the affordance doesn't matter as long as it exists. The annotations aren't great in many cases, hence why we also put a Human evaluation in the tables, to show that even a human cannot reach 100% when annotations aren't consistent. If you are using IoU, you might find performance dependent on the scale, so I suggest using the other metric from the paper as guide if it's correct or not.

from ucl-affcorrs.