Using LLMs and pre-trained caption models for super-human performance on image captioning.
License: Other
I joined Github 11 years ago.
Since then I've pushed 4206 commits, opened 50 issues, submitted 123 pull requests, received 216 stars across 102 personal projects and contributed to 17 public repositories.
Most used languages across my projects:
Generated using teoxoy/profile-readme-stats
I'm on Mastodon
Can we perform captioning tasks on pedestrian images in reality to obtain finer grained text descriptions.
I have some real-life pedestrian images and I would like to generate corresponding text descriptions for them. I am not sure if your model can be implemented, and I look forward to your reply!
Thank you!
Hello and thanks for your work. Would it be possible to share the generated output from the LLM for the COCO dataset?
trying to run it in cloud (paperspace gradient) but when running cbc caption test/test_image.jpg i get this:
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /notebooks/caption-by-committee/env/bin/cbc:5 in <module> │
│ │
│ 2 # -*- coding: utf-8 -*- │
│ 3 import re │
│ 4 import sys │
│ ❱ 5 from cbc.cli import main │
│ 6 if __name__ == '__main__': │
│ 7 │ sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) │
│ 8 │ sys.exit(main()) │
│ │
│ /notebooks/caption-by-committee/env/lib/python3.9/site-packages/cbc/cli.py:3 │
│ in <module> │
│ │
│ 1 import click │
│ 2 │
│ ❱ 3 from .caption_by_committee import caption │
│ 4 from .dataset import evaluate_dataset │
│ 5 from .plugins.test import test_plugin │
│ 6 │
│ │
│ /notebooks/caption-by-committee/env/lib/python3.9/site-packages/cbc/caption_ │
│ by_committee.py:10 in <module> │
│ │
│ 7 from PIL import Image │
│ 8 import logging │
│ 9 │
│ ❱ 10 from cbc.caption import CAPTION_ENGINES_CLI │
│ 11 from cbc.lm import ( │
│ 12 │ LM_ENGINES_CLI, │
│ 13 │ LM_LOCAL_ENGINES, │
│ │
│ /notebooks/caption-by-committee/env/lib/python3.9/site-packages/cbc/caption/ │
│ __init__.py:6 in <module> │
│ │
│ 3 from .base import CaptionEngine │
│ 4 from .blip_engine import BLIP2COCOBase, BLIP2COCOLarge, BLIP2COCOT5Larg │
│ 5 from .chatcaptioner_engine import ChatCaptionerEngine │
│ ❱ 6 from .ic3_engine import IC3CaptionEngine │
│ 7 from .ofa_engine import OFACaptionEngine │
│ 8 from .socratic_models import SocraticModelCaptionEngine │
│ 9 │
│ │
│ /notebooks/caption-by-committee/env/lib/python3.9/site-packages/cbc/caption/ │
│ ic3_engine.py:8 in <module> │
│ │
│ 5 from PIL import Image │
│ 6 │
│ 7 from cbc.caption.base import CaptionEngine │
│ ❱ 8 from cbc.caption.ic3.caption_by_committee import DEFAULT_CBC_PROMPT, ca │
│ 9 │
│ 10 │
│ 11 class IC3CaptionEngine(CaptionEngine): │
╰──────────────────────────────────────────────────────────────────────────────╯
ModuleNotFoundError: No module named 'cbc.caption.ic3'
followed instructions that state to install salesforce-lavis and then install the local directory (pip install .), but maybe the fact that the two require incompatible versions of the transformers package will cause an error? since i got this during install:
salesforce-lavis 1.0.2 requires transformers<4.27,>=4.25.0, but you have transformers 4.27.0 which is incompatible.
cbc 0.0.2 requires transformers>=4.28.0, but you have transformers 4.27.0 which is incompatible.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.