Gradio demo does not seem to work about caption-anything HOT 4 CLOSED

python app.py --regular_box --captioner blip2 --port 6086 --gradio_share --disable_reuse_features

works

from caption-anything.

the error on my end:

AssertionError
prompt: {'prompt_type': ['click'], 'input_point': [[2205, 1552], [2170, 1490], [1903, 1582], [1571, 995], [1663, 492]], 'input_label': [1, 1, 1, 1, 1], 'multimask_output': 'True'} controls: {'length': 10, 'sentiment': 'Natural', 'factuality': 'Factual', 'language': 'English'}
CA prompt: {'prompt_type': ['click'], 'input_point': [[2205, 1552], [2170, 1490], [1903, 1582], [1571, 995], [1663, 492]], 'input_label': [1, 1, 1, 1, 1], 'multimask_output': 'True'} CA controls {'length': 10, 'sentiment': 'Natural', 'factuality': 'Factual', 'language': 'English'}
Traceback (most recent call last):
File "/mnt/bd/data-tns-algo-us-audio-multimodal/environment/anaconda3/envs/python39/lib/python3.9/site-packages/gradio/routes.py", line 395, in run_predict
output = await app.get_blocks().process_api(
File "/mnt/bd/data-tns-algo-us-audio-multimodal/environment/anaconda3/envs/python39/lib/python3.9/site-packages/gradio/blocks.py", line 1193, in process_api
result = await self.call_function(
File "/mnt/bd/data-tns-algo-us-audio-multimodal/environment/anaconda3/envs/python39/lib/python3.9/site-packages/gradio/blocks.py", line 930, in call_function
prediction = await anyio.to_thread.run_sync(
File "/mnt/bd/data-tns-algo-us-audio-multimodal/environment/anaconda3/envs/python39/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/mnt/bd/data-tns-algo-us-audio-multimodal/environment/anaconda3/envs/python39/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/mnt/bd/data-tns-algo-us-audio-multimodal/environment/anaconda3/envs/python39/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/mnt/bd/data-tns-algo-us-audio-multimodal/environment/anaconda3/envs/python39/lib/python3.9/site-packages/gradio/utils.py", line 491, in async_iteration
return next(iterator)
File "/mnt/bd/data-tns-algo-us-audio-multimodal/experiment/Caption-Anything/app.py", line 115, in inference_seg_cap
out = model.inference(image_input, prompt, controls)
File "/mnt/bd/data-tns-algo-us-audio-multimodal/experiment/Caption-Anything/caption_anything.py", line 26, in inference
seg_mask = self.segmenter.inference(image, prompt)[0, ...]
File "/mnt/bd/data-tns-algo-us-audio-multimodal/environment/anaconda3/envs/python39/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "/mnt/bd/data-tns-algo-us-audio-multimodal/experiment/Caption-Anything/segmenter/base_segmenter.py", line 53, in inference
assert self.image_embedding is not None

from caption-anything.

The suggestions from GPT-4:

The AssertionError raised in the code is due to the fact that self.image_embedding is None. In the base_segmenter.py file, line 53 checks if self.image_embedding is not None, and raises an AssertionError if the condition fails:

assert self.image_embedding is not None

This error occurs because the self.image_embedding attribute of self.segmenter is not set before calling the inference method. In the app.py file, you are setting the image of the segmenter but not setting the image embedding:

model.segmenter.set_image(image_input)

In order to fix this issue, you need to ensure that the image_embedding attribute is properly set before calling the inference method.

First, check if the set_image implementation calculates and sets the image_embedding attribute in the segmenter. If not, you might need to adjust the implementation of the set_image method or create another method to set self.image_embedding for the segmenter.

Make sure to carefully review the entire codebase to see where and when self.image_embedding is supposed to be set, and ensure it is correctly calculated and assigned before the inference method is called.

from caption-anything.

Thank you for raising this issue.

The bug you encountered was due to the previous version of our code, which only extracted embeddings for user-uploaded images. The embeddings for example images were not extracted, causing an assertion error.

We have since resolved this issue. You can find the relevant details and fixes in this commit 28522a8. Please let us know if you have any further questions or concerns.

from caption-anything.