how to generate concept embedding file for new classes from a custom dataset? about regionclip HOT 8 CLOSED

Hi, @zhangxgu @drahmad89 , I had included the script to extract concept embeddings for customized concept pool, please have a check. Also, @zhangxgu , your code for extracting concept embeddings seems pretty correct except that you may want to use all prompt templates and do the normalization to the output text features.

from regionclip.

Hi, @drahmad89

You can refer to our demo code here:
https://huggingface.co/spaces/CVPR/regionclip-demo/blob/main/detectron2/modeling/meta_arch/clip_rcnn.py#L755

And you can build a text encoder here:
https://huggingface.co/spaces/CVPR/regionclip-demo/blob/main/detectron2/modeling/meta_arch/clip_rcnn.py#L593

from regionclip.

@jwyang
I wanted to get only bbox on specific classes.
in the documentation:" put it in the folder ./datasets/lvis/lvis_v1_val.json. The file is used to specify object class names."
I replaced this "lvis_v1_val.json" with custom annotation json file that only contains 6 different classes. I ran the zeroshot detection and always getting classes from LVIS dataset. (section:Visualization on custom images)

from regionclip.

@drahmad89 , currently, visualize on custom images does not support user-specific queries. We had built a huggingface demo as shared above which takes one category as the query. Per your request, we will build a customized demo into this repo to support user-specific queries given an image. Please stay tuned!

from regionclip.

Nice work! @jwyang
I also need the code to generate text embeddings of my own dataset. By now I write a code following CLIP like these:

import torch
import clip
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("RN50", device=device)

categories = ['XXX','XXX',XXX',]

text = clip.tokenize(["a photo of a %s"%c for c in categories]).to(device)

with torch.no_grad():
text_features = model.encode_text(text).cpu()

torch.save(text_features,'xxx.pth')

Hope you can give me some advice on this code.

from regionclip.

The script output at the end had a dim of (class_len, 1, emb_size), it is also still on gpu. I suggest to add torch.squeeze(concept_feats).cpu() after

I wasn't too sure of the format of the classes to input in concepts.txt. If it is listed line by line then shouldn't

for line in f:
            concept = line.strip()

replace

Just some suggestions :)

from regionclip.

Hi, @Jawing, good suggestion!

from regionclip.

There seems to be a bug with generating concept embeddings in extract_concept_features.py.

The generated embeddings seem to correspond to the first letter of each concepts, which can be duplicate and isn't what we want. I propose the correction below.

    concept_feats = []
    with open(concept_file, 'r') as f:
            concepts = []
            for line in f:
                concept = line.strip()
                concepts.append(concept)
            with torch.no_grad():
                token_embeddings_concepts = pre_tokenize(concepts).to(model.device)
                for token_embeddings in token_embeddings_concepts:
                    text_features = model.lang_encoder.encode_text(token_embeddings)
                    # average over all templates
                    text_features = text_features.mean(0, keepdim=True)
                    concept_feats.append(text_features)

from regionclip.