Comments (5)
Thank you for your interest in my work. I have uploaded the code for processing Next-QA with GLIP, which can be found in src/tools/README.md
and src/tools/extract_glip_bboxes.py
.
The item_dict['video_features']
refers to the CLIP's [cls]
token, and item_dict['bbox_features']
represents the features extracted by CLIP after GLIP's bbox extraction.
It's a bit cumbersome, but necessary since GLIP's ROI features cannot be directly extracted. Furthermore, even if extracted, they wouldn't align with CLIP's BERT representation.
from lgva_videoqa.
Thank you for your interest in my work. I have uploaded the code for processing Next-QA with GLIP, which can be found in
src/tools/README.md
andsrc/tools/extract_glip_bboxes.py
.The
item_dict['video_features']
refers to the CLIP's[cls]
token, anditem_dict['bbox_features']
represents the features extracted by CLIP after GLIP's bbox extraction.It's a bit cumbersome, but necessary since GLIP's ROI features cannot be directly extracted. Furthermore, even if extracted, they wouldn't align with CLIP's BERT representation.
Thank you for your reply. Isee that the extraxt_embedding.py loaded pre-train clip model to get an image_features of an image.
How to extract the region features of the generated bboxes from glip does not provide processing details.
Can you provide details on how to use clip to extract feature representations of regions for given bboxes?
from lgva_videoqa.
Using clip to extract features from the bbox region is straightforward. You can crop the original image using OpenCV or Pillow, and then use clip to extract features from the corresponding bbox region.
from lgva_videoqa.
hi, can you provide the code for extracting region features by clip after GLIP's bbox extraction?
from lgva_videoqa.
Sorry for the late response, I've been quite busy lately. Here is a simple example code:
x1, y1, x2, y2 = bbox
cropped_image = image_np[y1:y2, x1:x2]
cropped_image_pil = Image.fromarray(cropped_image)
tensor = val_transform(cropped_image_pil)
output = model.encode_image(tensor)
Recently, I've been too busy. If I find the time, I will organize and provide the complete code.
from lgva_videoqa.
Related Issues (12)
- Missing h5 extracted feature files HOT 2
- feature extraction flies HOT 2
- code reproduction problem HOT 2
- the meaning of qid in next_train_qa.json HOT 1
- the code for multi-gpu training HOT 1
- Incomplete extract_embedding.py code? HOT 1
- About CLIP’s text-encoder. HOT 3
- Processing of candidate words HOT 3
- Processing of GLIP HOT 2
- Clarification on Feature Representations from text_features_all.h5 and text_features_clip.h5 HOT 5
- About bbox_features in NeXt-clip-bbox-features.zip
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lgva_videoqa.