Data setting for zero-shot results about openscene HOT 4 CLOSED

Hi, @pengsongyou. I know the key in this paper is to construct a CLIP latent feature supervised 3D semantic backbone which has better generalization ability.
But I just wanted to clarify that the distillation training approach you are using is not a strict zero-shot segmentation setting. During distillation training, all points or pixels are used to train the 3D backbone regardless of whether they are seen or unseen categories. However, during zero-shot testing, you split the categories into seen and unseen, and only the unseen categories are used for evaluation.

In my opinion, in a zero-shot setting, the unseen points or pixels should not be used at all during training, regardless of the training strategy used.

regards,
zihui

from openscene.

Hi @ZhengLeon ,

During distillation, the only loss we have is the cosine similarity loss between the 2D fused features and the predicted per-point features. We do not really classify seen/unseen objects since we do not consider the object classes/labels at all. Please refer to section 3.2 in our paper.

Best
Songyou

from openscene.

Hi, @pengsongyou. I know the key in this paper is to construct a CLIP latent feature supervised 3D semantic backbone which has better generalization ability. But I just wanted to clarify that the distillation training approach you are using is not a strict zero-shot segmentation setting. During distillation training, all points or pixels are used to train the 3D backbone regardless of whether they are seen or unseen categories. However, during zero-shot testing, you split the categories into seen and unseen, and only the unseen categories are used for evaluation.

In my opinion, in a zero-shot setting, the unseen points or pixels should not be used at all during training, regardless of the training strategy used.

regards, zihui

Well, I think this is a good challenge. As it was discussed in "Definition of Zero-Shot Learning (ZSL)", supplementary, Sec. B. The ZSL setting followed in this paper is not quite strict. In real-world applications, however, train and inference on customized data without labeling can help a lot. Manpower consumption is more concerned.

from openscene.

@zhang-zihui

Yes, as @argosdh pointed out, we discussed this in the supplementary material. You can check it out. Basically, we follow the ZSL definition from the original CLIP paper, and many of its follow-up papers (e.g. LSeg, OpenSeg, etc), which is not strictly the same as the classic ZSL definition. Hope it clarifies your concern.

And sorry I missed your comments earlier and only replied now.

from openscene.