opengvlab / interngpt Goto Github PK

and also cannot open https://127.0.0.1:7862/ in browser

ctx.load_cert_chain(cerfile, keyfile, get_password)
FileNotFoundError: [Errorno 2] No such file or directory.

你能在线运行吗？而且，您将如何在Google Colab中运行它

只启动DragGAN,生成图片到了最后一步Error, 报错内容needs one of codec_name or template,请问该如何处理?

(igpt) [root@localhost InternGPT]# python -u app.py --load "StyleGAN_cuda:0" --tab "DragGAN" --port 3456 --https
[06/05 16:12:15] bark.generation WARNING: torch version does not support flash attention. You will get faster inference speed by upgrade torch to newest nightly version.
Initializing InternGPT, load_dict={'StyleGAN': 'cuda:0'}
Running on local URL: https://0.0.0.0:3456

To create a public link, set share=True in launch().
===>logging in
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
Traceback (most recent call last):
File "/usr/local/anaconda3/envs/igpt/lib/python3.8/site-packages/gradio/routes.py", line 399, in run_predict
output = await app.get_blocks().process_api(
File "/usr/local/anaconda3/envs/igpt/lib/python3.8/site-packages/gradio/blocks.py", line 1299, in process_api
result = await self.call_function(
File "/usr/local/anaconda3/envs/igpt/lib/python3.8/site-packages/gradio/blocks.py", line 1036, in call_function
prediction = await anyio.to_thread.run_sync(
File "/usr/local/anaconda3/envs/igpt/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/anaconda3/envs/igpt/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/usr/local/anaconda3/envs/igpt/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/usr/local/anaconda3/envs/igpt/lib/python3.8/site-packages/gradio/utils.py", line 488, in async_iteration
return next(iterator)
File "/data/InternGPT/iGPT/controllers/ConversationBot.py", line 920, in drag_it
imageio.mimsave(video_name, style_gan_state['history'])
File "/usr/local/anaconda3/envs/igpt/lib/python3.8/site-packages/imageio/v2.py", line 484, in mimwrite
return file.write(ims, is_batch=True, **kwargs)
File "/usr/local/anaconda3/envs/igpt/lib/python3.8/site-packages/imageio/plugins/pyav.py", line 634, in write
self.init_video_stream(codec, fps=fps, pixel_format=out_pixel_format)
File "/usr/local/anaconda3/envs/igpt/lib/python3.8/site-packages/imageio/plugins/pyav.py", line 846, in init_video_stream
stream = self._container.add_stream(codec, fps)
File "av/container/output.pyx", line 61, in av.container.output.OutputContainer.add_stream
ValueError: needs one of codec_name or template

Hello. I somehow managed to get the LLama weights manually as per the directory structure. Everything seemed to be going fine. However, when the Husky model started to load, I got this error:

OSError: model_zoo/husky-7b-delta-v0_01 is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.

From further inspection, it seems that that the Husky 7B model is not available on Hugging Face. Any help on this front is appreciated.

羽毛球的制胜规律

when running this command

python -u app.py \ --load "ImageOCRRecognition_cuda:0,Text2Image_cuda:0,SegmentAnything_cuda:0,ActionRecognition_cuda:0,VideoCaption_cuda:0,DenseCaption_cuda:0,ReplaceMaskedAnything_cuda:0,LDMInpainting_cuda:0,SegText2Image_cuda:0,ScribbleText2Image_cuda:0,Image2Scribble_cuda:0,Image2Canny_cuda:0,CannyText2Image_cuda:0,StyleGAN_cuda:0,Anything2Image_cuda:0,HuskyVQA_cuda:0" -e -p 3456 --https

I met this error

Are there image to audio capabilities by any chance?

Hi,

Videos of InternGPT looks highly potential, but we stuck up loading the models with the following error:
"OSError: model_zoo/llama_7B_hf does not appear to have a file named config.json"

We understand that it can be due to the license issue. For original checkpoint of LLAMA, we also requested by filling google form but unfortunately we didn't hear any from Meta yet.

Hence can you please suggest if there is any other way to remove dependency on LLaMA and what could be the compatible model to use with InternGPT otherwise?

Thank you in advance.

HI, how is the experiment in Table 4 conducted? What if we have a large dataset e.g. 1000 VQA samples, how could we conduct the experiment?

conda install pytorch==1.13.0 torchvision==0.14.0 torchaudio==0.13.0 pytorch-cuda=11.6 -c pytorch -c nvidia
Collecting package metadata (current_repodata.json): - WARNING conda.models.version:get_matcher(535): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 1.7.1.*, but conda is ignoring the .* and treating it as 1.7.1
done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): | WARNING conda.models.version:get_matcher(535): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 1.9.0.*, but conda is ignoring the .* and treating it as 1.9.0
WARNING conda.models.version:get_matcher(535): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 1.8.0.*, but conda is ignoring the .* and treating it as 1.8.0
WARNING conda.models.version:get_matcher(535): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 1.6.0.*, but conda is ignoring the .* and treating it as 1.6.0
done
Solving environment: - 

I wonder if you ever got stuck here before, thanks!

Hi~
Thanks for your great work!

I have read your paper and went through in detail this script (https://github.com/OpenGVLab/InternGPT/blob/main/iGPT/controllers/ConversationBot.py).

I noticed that the used visual models are determined by some key words, i.e., remove & erase means LDMInpainting, describe & introduce means HuskyVQA. This is a direct and effective way.

However, I wonder what will happen if the user does not input such words. For example, the user could input take out some objects instead of remove some objects for object removing.

Thanks in advance.

开VPN 时候报错如下：
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError(0, 'Error'))': /simple/draggan/
不开VPN 下载一会报错如下：
下载大概不到一分钟，就速度为0 ，就报错如下
ERROR: Exception:
Traceback (most recent call last):
File "C:\Users\Administrator.conda\envs\draggan\lib\site-packages\pip_vendor\urllib3\response.py", line 437, in _error_catcher
yield
File "C:\Users\Administrator.conda\envs\draggan\lib\site-packages\pip_vendor\urllib3\response.py", line 560, in read
data = self._fp_read(amt) if not fp_closed else b""
File "C:\Users\Administrator.conda\envs\draggan\lib\site-packages\pip_vendor\urllib3\response.py", line 526, in _fp_read
return self._fp.read(amt) if amt is not None else self._fp.read()
File "C:\Users\Administrator.conda\envs\draggan\lib\site-packages\pip_vendor\cachecontrol\filewrapper.py", line 90, in read
data = self.__fp.read(amt)

我一直在尝试让这个项目正常运行，但一直遇到很多问题。请问有没有可能提供一个Docker文件来容器化这个项目？
I have been trying to make this project work properly, but I have encountered many problems. Is it possible to provide a Docker file to containerize this project?

在gradio的demo上不管跑什么模块，终端都报同样的错误。
Traceback (most recent call last):
File "/mnt/ai212/workspace/dengtb/anaconda3/envs/ichat/lib/python3.8/site-packages/gradio/routes.py", line 414, in run_predict
output = await app.get_blocks().process_api(
File "/mnt/ai212/workspace/dengtb/anaconda3/envs/ichat/lib/python3.8/site-packages/gradio/blocks.py", line 1323, in process_api
data = self.postprocess_data(fn_index, result["prediction"], state)
File "/mnt/ai212/workspace/dengtb/anaconda3/envs/ichat/lib/python3.8/site-packages/gradio/blocks.py", line 1257, in postprocess_data
prediction_value = block.postprocess(prediction_value)
File "/mnt/ai212/workspace/dengtb/anaconda3/envs/ichat/lib/python3.8/site-packages/gradio/components.py", line 4629, in postprocess
assert isinstance(
AssertionError: Expected a list of lists or list of tuples. Received: None

Hi! I noticed that you may wrongly type 'marked' as 'maked' in the Chinese doc.

According to the term of use https://openai.com/policies/terms-of-use, users shouldn't share their api key with any organization.

Hello, just a heads up warning:
I think it's looking for the model in two different locations

• model_zoo/llama\7B\
• model_zoo\llama_7B_hf
  If I copy the model to both locations the demo server comes up.

(ichat) E:\ai\InternGPT>python -u app.py --load "HuskyVQA_cuda:0,SegmentAnything_cuda:0,ImageOCRRecognition_cuda:0" --port 3456
[05/17 20:42:38] bark.generation WARNING: torch version does not support flash attention. You will get faster inference speed by upgrade torch to newest nightly version.
Initializing InternGPT, load_dict={'HuskyVQA': 'cuda:0', 'SegmentAnything': 'cuda:0', 'ImageOCRRecognition': 'cuda:0'}
Für das Windows-Subsystem für Linux wurden keine Distributionen installiert.
Distributionen zur Installation finden Sie im Microsoft Store:
https://aka.ms/wslstore
Traceback (most recent call last):
  File "app.py", line 221, in <module>
    bot = ConversationBot(load_dict=load_dict)
  File "E:\ai\InternGPT\iGPT\controllers\ConversationBot.py", line 141, in __init__
    self.models[class_name] = globals()[class_name](device=device)
  File "E:\ai\InternGPT\iGPT\models\husky.py", line 368, in __init__
    download_if_not_exists(base_path="model_zoo/llama",
  File "E:\ai\InternGPT\iGPT\models\husky.py", line 351, in download_if_not_exists
    write_model(
  File "E:\ai\InternGPT\iGPT\models\husky_src\convert_llama_weights_to_hf.py", line 93, in write_model
    params = read_json(os.path.join(input_base_path, "params.json"))
  File "E:\ai\InternGPT\iGPT\models\husky_src\convert_llama_weights_to_hf.py", line 79, in read_json
    with open(path, "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'model_zoo/llama\\7B\\params.json'

(ichat) E:\ai\InternGPT>python -u app.py --load "HuskyVQA_cuda:0,SegmentAnything_cuda:0,ImageOCRRecognition_cuda:0" --port 3456
[05/17 20:44:16] bark.generation WARNING: torch version does not support flash attention. You will get faster inference speed by upgrade torch to newest nightly version.
Initializing InternGPT, load_dict={'HuskyVQA': 'cuda:0', 'SegmentAnything': 'cuda:0', 'ImageOCRRecognition': 'cuda:0'}
Loading base model
Traceback (most recent call last):
  File "app.py", line 221, in <module>
    bot = ConversationBot(load_dict=load_dict)
  File "E:\ai\InternGPT\iGPT\controllers\ConversationBot.py", line 141, in __init__
    self.models[class_name] = globals()[class_name](device=device)
  File "E:\ai\InternGPT\iGPT\models\husky.py", line 368, in __init__
    download_if_not_exists(base_path="model_zoo/llama",
  File "E:\ai\InternGPT\iGPT\models\husky.py", line 359, in download_if_not_exists
    apply_delta(output_dir, new_path, delta_path)
  File "E:\ai\InternGPT\iGPT\models\husky_src\load_ckpt.py", line 11, in apply_delta
    base = AutoModelForCausalLM.from_pretrained(base_model_path, torch_dtype=torch.float16, low_cpu_mem_usage=True)
  File "C:\Users\Sasch\.conda\envs\ichat\lib\site-packages\transformers\models\auto\auto_factory.py", line 441, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
  File "C:\Users\Sasch\.conda\envs\ichat\lib\site-packages\transformers\models\auto\configuration_auto.py", line 916, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "C:\Users\Sasch\.conda\envs\ichat\lib\site-packages\transformers\configuration_utils.py", line 573, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "C:\Users\Sasch\.conda\envs\ichat\lib\site-packages\transformers\configuration_utils.py", line 628, in _get_config_dict
    resolved_config_file = cached_file(
  File "C:\Users\Sasch\.conda\envs\ichat\lib\site-packages\transformers\utils\hub.py", line 380, in cached_file
    raise EnvironmentError(
OSError: model_zoo\llama_7B_hf does not appear to have a file named config.json. Checkout 'https://huggingface.co/model_zoo\llama_7B_hf/None' for available files.

Error after running: python app.py

File "/Users/username/Development/InternGPT/InternGPT/iGPT/models/image.py", line 20, in
from controlnet_aux import OpenposeDetector, MLSDdetector, HEDdetector
ModuleNotFoundError: No module named 'controlnet_aux'

After installation according to guidiance, i run the follow commands, then the error happen.
python -u app.py --load "HuskyVQA_cuda:0,SegmentAnything_cuda:0,ImageOCRRecognition_cuda:0" --port 3456

Looking in indexes: http://mirrors.ops.ctripcorp.com/pypi-latest/simple
Collecting git+https://github.com/facebookresearch/detectron2.git
Cloning https://github.com/facebookresearch/detectron2.git to /tmp/pip-req-build-tcy8m3_g
Running command git clone --filter=blob:none --quiet https://github.com/facebookresearch/detectron2.git /tmp/pip-req-build-tcy8m3_g
Resolved https://github.com/facebookresearch/detectron2.git to commit 3c7bb714795edc7a96c9a1a6dd83663ecd293e36
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

_ python setup.py egg_info did not run successfully.
_ exit code: 1
__> [12 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/tmp/pip-req-build-tcy8m3_g/setup.py", line 10, in
import torch
File "/home/powerop/work/conda/envs/igpt/lib/python3.8/site-packages/torch/init.py", line 191, in
_load_global_deps()
File "/home/powerop/work/conda/envs/igpt/lib/python3.8/site-packages/torch/init.py", line 153, in _load_global_deps
ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
File "/home/powerop/work/conda/envs/igpt/lib/python3.8/ctypes/init.py", line 373, in init
self._handle = _dlopen(self._name, mode)
OSError: /home/powerop/work/conda/envs/igpt/lib/python3.8/site-packages/torch/lib/../../nvidia/cublas/lib/libcublas.so.11: symbol cublasLtHSHMatmulAlgoInit version libcublasLt.so.11 not defined in file libcublasLt.so.11 with link time reference
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

_ Encountered error while generating package metadata.
__> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Traceback (most recent call last):
File "app.py", line 28, in
from iGPT.controllers import ConversationBot
File "/home/powerop/work/gongsong/InternGPT/iGPT/init.py", line 1, in
from .models import *
File "/home/powerop/work/gongsong/InternGPT/iGPT/models/init.py", line 1, in
from .image import (InstructPix2Pix, ImageText2Image,
File "/home/powerop/work/gongsong/InternGPT/iGPT/models/image.py", line 2, in
import torch
File "/home/powerop/work/conda/envs/igpt/lib/python3.8/site-packages/torch/init.py", line 191, in
_load_global_deps()
File "/home/powerop/work/conda/envs/igpt/lib/python3.8/site-packages/torch/init.py", line 153, in _load_global_deps
ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
File "/home/powerop/work/conda/envs/igpt/lib/python3.8/ctypes/init.py", line 373, in init
self._handle = _dlopen(self._name, mode)
OSError: /home/powerop/work/conda/envs/igpt/lib/python3.8/site-packages/torch/lib/../../nvidia/cublas/lib/libcublas.so.11: symbol cublasLtHSHMatmulAlgoInit version libcublasLt.so.11 not defined in file libcublasLt.so.11 with link time reference

Background

Dependencies in requirements.txt have module conflicts.

Description

There are two dependencies mentioned in the requirements.txt file: opencv-python and albumentations and the 'albumentations' depends on opencv-python-headless. The official spec mentioned that the opencv-python package is for the desktop environment, while opencv-python-headless is for the server environment. The documentation also states that these two packages cannot be installed simultaneously (the exact wording is: “There are four different packages (see options 1, 2, 3, and 4 below) and you should SELECT ONLY ONE OF THEM.”). This is because they both use the same module name cv2.

During the installation process using pip, the package installed later will override the cv2 module from the previously installed package (specifically, the modules within the cv2 folders that exist in both packages). Furthermore, the dependency graph even includes different versions of these two packages. It is certain that the common files with the same path in these two packages contain different contents. Therefore, there may be functional implications when using them. However, without analyzing the specific code and function call hierarchy of this project, it can be stated that issues related to overwriting and module conflicts do exist.

Steps to Reproduce

pip install -r requirements.txt

Desired Change

Indeed, it is not an ideal behavior for modules to be overwritten, even if they are not actively used or if the overwritten module is the one being called. It introduces uncertainty and can cause issues in the long run, especially if there are changes or updates to the overwritten modules in future development. It is generally recommended to avoid such conflicts and ensure that only the necessary and compatible dependencies are declared in the requirements to maintain a stable and predictable environment for the project.

We believe that although this project can only modify direct dependencies and indirect dependencies are a black box, it is possible to add additional explanations rather than directly declaring both conflicting packages in the requirements.txt file.

Adding extra explanations or documentation about the potential conflicts and the need to choose only one of the conflicting packages can help developers understand the issue and make informed decisions. Including a clear instruction or warning in the project’s documentation can guide users to choose the appropriate package based on their specific requirements.

Hello. I have been trying to setup InternGPT with local LLama2 downloads. But even after entering the proper link and checking multiple times, I am getting this.

Initializing InternGPT, load_dict={'HuskyVQA': 'cuda:0'}
Downloading tokenizer
model_zoo/llama/tokenizer.model: No such file or directory
model_zoo/llama/tokenizer_checklist.chk: No such file or directory
third-party/llama_download.sh: line 19: cd: model_zoo/llama: No such file or directory
Downloading 7B
--2023-11-18 16:15:15--  https://download.llamameta.net/7B/consolidated.00.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoicjlpeHBvdDBoZHVlanRwdHB1dDhqZDA1IiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMDM4OTg0N319fV19&Signature=lL2YWTPbU8BSWD0wQfqPZ2cjvur44OSSkSXe63V7rvBpCZ80I%7EivDgMaay%7E8dYiOXNj6ULoJJE-Tyl6xn51AW4etc6bP1p2anPc3pWCd-q48GKyKYyvVvOR44EOxfp9dSVzuUBMk83VXkILzGn7kDUWYooGWrov3kRSK72-d2zhsPdcYtdVijc1rG%7EUorXDz8pkUDHUeOHNxgOCQL-0WN-u8BDlvH2HFAbJLWSl1M-Gi4rR4wkyxjH%7EmTqdt-qmaob5L1lF6N9D1jCTupNnIzYMDxBb7sz5qvp6OlBwJonMYGu2tlN%7Ea4DLNT7a-3aHF2JPGLoilVKVt8XexfaTJ4A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=997730767997515
Resolving download.llamameta.net (download.llamameta.net)... 18.154.144.23, 18.154.144.95, 18.154.144.56, ...
Connecting to download.llamameta.net (download.llamameta.net)|18.154.144.23|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-11-18 16:15:17 ERROR 403: Forbidden.

--2023-11-18 16:15:17--  https://download.llamameta.net/7B/params.json?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoicjlpeHBvdDBoZHVlanRwdHB1dDhqZDA1IiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMDM4OTg0N319fV19&Signature=lL2YWTPbU8BSWD0wQfqPZ2cjvur44OSSkSXe63V7rvBpCZ80I%7EivDgMaay%7E8dYiOXNj6ULoJJE-Tyl6xn51AW4etc6bP1p2anPc3pWCd-q48GKyKYyvVvOR44EOxfp9dSVzuUBMk83VXkILzGn7kDUWYooGWrov3kRSK72-d2zhsPdcYtdVijc1rG%7EUorXDz8pkUDHUeOHNxgOCQL-0WN-u8BDlvH2HFAbJLWSl1M-Gi4rR4wkyxjH%7EmTqdt-qmaob5L1lF6N9D1jCTupNnIzYMDxBb7sz5qvp6OlBwJonMYGu2tlN%7Ea4DLNT7a-3aHF2JPGLoilVKVt8XexfaTJ4A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=997730767997515
Resolving download.llamameta.net (download.llamameta.net)... 18.154.144.23, 18.154.144.95, 18.154.144.56, ...
Connecting to download.llamameta.net (download.llamameta.net)|18.154.144.23|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-11-18 16:15:18 ERROR 403: Forbidden.

--2023-11-18 16:15:18--  https://download.llamameta.net/7B/checklist.chk?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoicjlpeHBvdDBoZHVlanRwdHB1dDhqZDA1IiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMDM4OTg0N319fV19&Signature=lL2YWTPbU8BSWD0wQfqPZ2cjvur44OSSkSXe63V7rvBpCZ80I%7EivDgMaay%7E8dYiOXNj6ULoJJE-Tyl6xn51AW4etc6bP1p2anPc3pWCd-q48GKyKYyvVvOR44EOxfp9dSVzuUBMk83VXkILzGn7kDUWYooGWrov3kRSK72-d2zhsPdcYtdVijc1rG%7EUorXDz8pkUDHUeOHNxgOCQL-0WN-u8BDlvH2HFAbJLWSl1M-Gi4rR4wkyxjH%7EmTqdt-qmaob5L1lF6N9D1jCTupNnIzYMDxBb7sz5qvp6OlBwJonMYGu2tlN%7Ea4DLNT7a-3aHF2JPGLoilVKVt8XexfaTJ4A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=997730767997515
Resolving download.llamameta.net (download.llamameta.net)... 18.154.144.23, 18.154.144.95, 18.154.144.56, ...
Connecting to download.llamameta.net (download.llamameta.net)|18.154.144.23|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-11-18 16:15:18 ERROR 403: Forbidden.

Is anyone else facing this issue?

How to request the download URL for the llama weights?

It seems this has a dependency on LLaMa base weights and Segment Anything. Is that correct?

Is there a way to separate this out and use OpenAI or an actually commercially viable model such as MosaicML's MPT-7B-Instruct?

Hi,

When can we expect the VisionLLM code to be released here?

How can I run it online? I've tried Colab, Gradio, etc...I know there's a online demo, but it is suspended, and I can't use it and I can't wait, also, my PC doesnt meet the requirements, so if someone can provide the code, give it!

What this 461M file is? Seems that it is not in the model_zoo. And I cannot find the url to download it manualy.

root@autodl-container-895011b752-eff81c3b:~/autodl-tmp/InternGPT_github/InternGPT# python -u app.py --load "StyleGAN_cuda:0" --tab "DragGAN" --port 19991 --https -e
[08/23 10:29:05] bark.generation WARNING: torch version does not support flash attention. You will get faster inference speed by upgrade torch to newest nightly version.
Initializing InternGPT, load_dict={'StyleGAN': 'cuda:0'}
 11%|████▏                                | 51.8M/461M [09:00<1:10:47, 101kiB/s]

给的网站打不开
https://igpt.opengvlab.com

After the demo server spins up, I cannot login to OpenAI with an api key. I confirmed with another app that the api key is good.
In the GUI I see:

Incorrect key, please input again

The console just logs:
===>logging in

How can I debug this further?

When I run python -u app.py --load "StyleGAN_cuda:0" --tab "DragGAN" --port 3456 --https -e, the web page requires an OpenAI API key to log in. Is the connection to World Wide Web essential for running your works?

论文中没有说明Husky的训练过程，只说了是三阶段的训练？可以详细说一下这三阶段训练的过程与用到的数据集吗

🫡

I get the following error:

(igpt) host:~/fa/InternGPT$ python -u app.py --load "ImageOCRRecognition_cuda:0,Text2Image_cuda:0,SegmentAnything_cuda:0,ActionRecognition_cuda:0,VideoCaption_cuda:0,DenseCaption_cuda:0,ReplaceMaskedAnything_cuda:0,LDMInpainting_cuda:0,SegText2Image_cuda:0,ScribbleText2Image_cuda:0,Image2Scribble_cuda:0,Image2Canny_cuda:0,CannyText2Image_cuda:0,StyleGAN_cuda:0,Anything2Image_cuda:0,HuskyVQA_cuda:0" -e -p 3456 --https
[07/19 11:24:32] bark.generation WARNING: torch version does not support flash attention. You will get faster inference speed by upgrade torch to newest nightly version.
Initializing InternGPT, load_dict={'ImageOCRRecognition': 'cuda:0', 'Text2Image': 'cuda:0', 'SegmentAnything': 'cuda:0', 'ActionRecognition': 'cuda:0', 'VideoCaption': 'cuda:0', 'DenseCaption': 'cuda:0', 'ReplaceMaskedAnything': 'cuda:0', 'LDMInpainting': 'cuda:0', 'SegText2Image': 'cuda:0', 'ScribbleText2Image': 'cuda:0', 'Image2Scribble': 'cuda:0', 'Image2Canny': 'cuda:0', 'CannyText2Image': 'cuda:0', 'StyleGAN': 'cuda:0', 'Anything2Image': 'cuda:0', 'HuskyVQA': 'cuda:0'}
Initializing ImageOCRRecognition to cuda:0
Initializing Text2Image to cuda:0
text_config_dict is provided which will be used to initialize CLIPTextConfig. The value text_config["id2label"] will be overriden.
Initializing SegmentAnything to cuda:0
Traceback (most recent call last):
File "app.py", line 225, in
bot = ConversationBot(load_dict=load_dict, e_mode=args.e_mode)
File "/home/ubuntu/fa/InternGPT/iGPT/controllers/ConversationBot.py", line 144, in init
self.models[class_name] = globals()class_name
File "/home/ubuntu/fa/InternGPT/iGPT/models/image.py", line 672, in init
self.sam = sam_model_registrymodel_type
File "/home/ubuntu/anaconda3/envs/igpt/lib/python3.8/site-packages/segment_anything/build_sam.py", line 15, in build_sam_vit_h
return _build_sam(
File "/home/ubuntu/anaconda3/envs/igpt/lib/python3.8/site-packages/segment_anything/build_sam.py", line 105, in _build_sam
state_dict = torch.load(f)
File "/home/ubuntu/anaconda3/envs/igpt/lib/python3.8/site-packages/torch/serialization.py", line 795, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/ubuntu/anaconda3/envs/igpt/lib/python3.8/site-packages/torch/serialization.py", line 1002, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.

Traceback (most recent call last):
File "/mnt/data/creative/InternGPT/app.py", line 225, in
bot = ConversationBot(load_dict=load_dict, e_mode=args.e_mode)
File "/mnt/data/creative/InternGPT/iGPT/controllers/ConversationBot.py", line 144, in init
self.models[class_name] = globals()class_name
File "/mnt/data/creative/InternGPT/iGPT/models/husky.py", line 369, in init
download_if_not_exists(base_path="model_zoo/llama",
File "/mnt/data/creative/InternGPT/iGPT/models/husky.py", line 359, in download_if_not_exists
apply_delta(output_dir, new_path, delta_path)
File "/mnt/data/creative/InternGPT/iGPT/models/husky_src/load_ckpt.py", line 11, in apply_delta
base = AutoModelForCausalLM.from_pretrained(base_model_path, torch_dtype=torch.float16,from_tf=True,low_cpu_mem_usage=True)
File "/mnt/data/creative/miniconda3/envs/internGPT/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 471, in from_pretrained
return model_class.from_pretrained(
File "/mnt/data/creative/miniconda3/envs/internGPT/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2753, in from_pretrained
if resolved_archive_file.endswith(".index"):
AttributeError: 'list' object has no attribute 'endswith'

When run python -u app.py --load "HuskyVQA_cuda:1,SegmentAnything_cuda:2,ImageOCRRecognition_cuda:3" --port 7863 -e，it happens
How to solve it？

微信群二维码已过期，求更新

where is certificate folder?

OSError: Unable to load weights from pytorch checkpoint file for 'model_zoo/llama_7B_hf/pytorch_model-00002-of-00033.bin' at 'model_zoo/llama_7B_hf/pytorch_model-00002-of-00033.bin'. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.
When run python -u app.py --load "HuskyVQA_cuda:1,SegmentAnything_cuda:2,ImageOCRRecognition_cuda:3" --port 7863 -e，it happens
How to solve it？