mbzuai-oryx / xraygpt Goto Github PK
View Code? Open in Web Editor NEW[BIONLP@ACL 2024] XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models.
[BIONLP@ACL 2024] XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models.
Hi,
great work! thanks for publishing you code and data.
What exactly do the radiology conversations in Healthcare_Radiology_vicuna.json contain? It still seems to be related to the data from ChatDoctor - did you somehow filter this data or how did you generate the radiology conversation data?
Thanks!
Hello,
Thank you for your wonderful work and I would like to ask you:
How to get High Quality and Interactive Summary Xray Datasets?
Maybe it can help us with some other work.
Thank you!
Are you only training the linear projection layer? In any stage are you training Qformer layers?
I’ve tried both requirements.txt as well as the .env files. Both are failing. I’ve made multiple changes as well.
I'm facing the same issue as well. I'm using Ubuntu 22.04.04 LTS. I modifed the @file modifier. It was giving some or other dependencies issues.
accelerate==0.15.0 aiofiles==23.1.0 aiohttp==3.8.4 aiosignal==1.3.1 albumentations==1.3.0 altair==4.2.2 antlr4-python3-runtime==4.9.3 anyio==3.6.2 appdirs==1.4.4 argon2-cffi==21.3.0 argon2-cffi-bindings==21.2.0 arrow==1.2.3 asttokens==2.2.1 async-timeout==4.0.2 attrs==22.2.0 backcall==0.2.0 beautifulsoup4==4.12.2 bitsandbytes==0.37.0 bleach==6.0.0 blis==0.7.9 braceexpand==0.1.7 brotlipy==0.7.0 cachetools==5.3.0 catalogue==2.0.8 cchardet==2.1.7 certifi cffi chardet==3.0.4 charset-normalizer cmake==3.26.3 comm==0.1.3 confection==0.0.4 contourpy==1.0.7 cryptography cycler==0.11.0 cymem==2.0.7 dataclasses==0.6 datasets debugpy==1.6.7 decorator==5.1.1 decord==0.6.0 defusedxml==0.7.1 dill docker-pycreds==0.4.0 entrypoints==0.4 et-xmlfile==1.1.0 evaluate==0.4.0 executing==1.2.0 ExifRead-nocycle==3.0.1 fairscale==0.4.13 fastapi==0.95.1 fastChat==0.1 fastjsonschema==2.16.3 ffmpy==0.3.0 filelock==3.9.0 fire flit_core fonttools==4.38.0 fqdn==1.5.1 frozenlist==1.3.3 fschat @ git+https://github.com/lm-sys/FastChat.git@f34f28cedcb8906fd026f22ec3ef41435a8e24ac fsspec gensim==4.3.1 gitdb==4.0.10 GitPython==3.1.31 googletrans==3.0.0 gradio==3.23.0 gradio_client==0.0.8 h11==0.9.0 h2==3.2.0 hiq-python==1.1.12 hpack==3.0.0 hstspreload==2023.1.1 httpcore==0.9.1 httpx==0.13.3 huggingface-hub hyperframe==5.2.0 idna==2.10 imageio==2.27.0 img2dataset importlib-metadata==6.5.0 importlib-resources==5.12.0 iopath==0.1.10 ipykernel==6.22.0 ipython==8.12.0 ipython-genutils==0.2.0 isoduration==20.11.0 jedi==0.18.2 Jinja2==3.1.2 joblib==1.2.0 jsonpointer==2.3 jsonschema==4.17.3 jupyter-events==0.6.3 jupyter_client==8.2.0 jupyter_core==5.3.0 jupyter_server==2.5.0 jupyter_server_terminals==0.4.4 jupyterlab-pygments==0.2.2 kiwisolver==1.4.4 langcodes==3.3.0 lazy_loader==0.2 linkify-it-py==2.0.0 lit==16.0.1 llvmlite==0.39.1 markdown-it-py==2.2.0 markdown2==2.4.8 MarkupSafe==2.1.2 matplotlib==3.7.0 matplotlib-inline==0.1.6 mdit-py-plugins==0.3.3 mdurl==0.1.2 MedCLIP==0.0.3 mistune==2.0.5 mkl-fft mkl-random mkl-service==2.4.0 mpmath==1.3.0 multidict==6.0.4 multiprocess murmurhash==1.0.9 nbclassic==0.5.5 nbclient==0.7.3 nbconvert==7.3.1 nbformat==5.8.0 nest-asyncio==1.5.6 networkx==3.1 nltk==3.8.1 notebook==6.5.4 notebook_shim==0.2.2 numba numpy nvidia-cublas-cu11==11.10.3.66 nvidia-cuda-cupti-cu11==11.7.101 nvidia-cuda-nvrtc-cu11==11.7.99 nvidia-cuda-runtime-cu11==11.7.99 nvidia-cudnn-cu11==8.5.0.96 nvidia-cufft-cu11==10.9.0.58 nvidia-curand-cu11==10.2.10.91 nvidia-cusolver-cu11==11.4.0.1 nvidia-cusparse-cu11==11.7.4.91 nvidia-nccl-cu11==2.14.3 nvidia-nvtx-cu11==11.7.91 omegaconf==2.3.0 openai==0.27.0 opencv-python==4.7.0.72 opencv-python-headless==4.7.0.72 openpyxl==3.1.2 orjson==3.8.10 packaging==23.0 pandas==1.5.3 pandocfilters==1.5.0 parso==0.8.3 pathtools==0.1.2 pathy==0.10.1 peft==0.2.0 pexpect==4.8.0 pickleshare==0.7.5 Pillow==9.4.0 platformdirs==3.2.0 portalocker==2.7.0 preshed==3.0.8 prometheus-client==0.16.0 promise==2.3 prompt-toolkit==3.0.38 protobuf==3.20.3 psutil==5.9.4 ptyprocess==0.7.0 pure-eval==0.2.2 py-itree==0.0.19 pyarrow pycocoevalcap==1.2 pycocotools==2.0.6 pycparser pydantic==1.10.7 pydub==0.25.1 Pygments==2.15.1 pyllama==0.0.9 pynndescent==0.5.9 pyOpenSSL pyparsing==3.0.9 pyrsistent==0.19.3 PySocks python-dateutil==2.8.2 python-json-logger==2.0.7 python-multipart==0.0.6 pytz==2023.3 PyWavelets==1.4.1 PyYAML==6.0 pyzmq==25.0.2 qudida==0.0.4 regex==2022.10.31 requests responses==0.18.0 rfc3339-validator==0.1.4 rfc3986==1.5.0 rfc3986-validator==0.1.1 rich==13.3.4 scikit-image==0.20.0 scikit-learn==1.2.2 scipy semantic-version==2.10.0 Send2Trash==1.8.0 sentence-transformers sentencepiece==0.1.97 sentry-sdk==1.19.1 setproctitle==1.3.2 shortuuid==1.0.11 six smart-open==6.3.0 smmap==5.0.0 sniffio==1.3.0 soupsieve==2.4.1 spacy==3.5.1 spacy-legacy==3.0.12 spacy-loggers==1.0.4 srsly==2.4.6 stack-data==0.6.2 starlette==0.26.1 svgwrite==1.4.3 sympy==1.11.1 tenacity==8.2.2 termcolor==2.2.0 terminado==0.17.1 textaugment==1.3.4 textblob==0.17.1 thinc==8.1.9 threadpoolctl==3.1.0 tifffile==2023.4.12 timm==0.6.13 tinycss2==1.2.1 tokenizers==0.13.2 toolz==0.12.0 torch==2.0.0 torchaudio torchvision tornado==6.3 tqdm traitlets==5.9.0 transformers triton==2.0.0 typer==0.7.0 typing_extensions tzdata==2023.3 uc-micro-py==1.0.1 umap-learn==0.5.3 uri-template==1.2.0 urllib3 uvicorn==0.21.1 wandb==0.12.21 wasabi==1.1.1 wavedrom==2.0.3.post3 wcwidth==0.2.6 webcolors==1.13 webdataset==0.2.48 webencodings==0.5.1 websocket-client==1.5.1 websockets==11.0.2 wget==3.2 xxhash==3.2.0 yarl==1.8.2 zipp==3.14.0
Currently using this. Getting numpy dependency error with mkl-fft and mkl-random. Can you kindly give me a .env or requirements.txt file.
/root/autodl-tmp/XrayGLM/chatglm-6b tokenizer! Now you can get tokenizer() everywhere
其中ChatGLM-6B中的tokenizer()需要全部的权重吗?
Dear authors,
I have some questions about the paper content:
(1) what is the MedVicuna and RadVicuna in Table 1? I cannot find them in the paper or on the Internet;
(2) According to Figure 1, it seems only the Linear Transformation Layer is trained in the whole framework, but why you mentioned in the contributions that "The LLM (Vicuna) is fine-turned on medical data"?
(3) In your work, if only the Linear Transformation Layer is trained while the LLM and MedClip are all frozen?
Dear Authors,
Thanks for your work on this project. I'm really interested in this work. I'm wondering have you done any more experiments on your generated reports? I would very much appreciate it if you could upload further results of your model.
Best!
He kept answering me in English when I asked questions in the make it language. So what do you think about using other languages for training in fintune rather than pre-training the whole model?
Hello, thank you for sharing your data and code, but I don't know if you missed the pre-trained vicuna_weights
/home/jgzn/anaconda/envs/XrayGpt/lib/python3.9/site-packages/transformers/generation/utils.py:1255: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)
warnings.warn(
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/jgzn/PycharmProjects/Xray/XrayGPT-main/demo1.py:69 in │
│ │
│ 66 │
│ 67 question = "Could you provide a detailed description of the given x-ray │
│ 68 chat.ask(question, chat_state) │
│ ❱ 69 answer, output_token = chat.answer(chat_state, img_list, num_beams=1, │
│ 70 │ │ │ │ │ │ │ │ temperature=1, │
│ 71 │ │ │ │ │ │ │ │ max_new_tokens=300, │
│ 72 │ │ │ │ │ │ │ │ max_length=2000) │
│ │
│ /home/jgzn/PycharmProjects/Xray/XrayGPT-main/xraygpt/conversation/conversati │
│ on.py:163 in answer │
│ │
│ 160 │ │ │
│ 161 │ │ embs = embs[:, begin_idx:] │
│ 162 │ │ │
│ ❱ 163 │ │ outputs = self.model.llama_model.generate( │
│ 164 │ │ │ inputs_embeds=embs, │
│ 165 │ │ │ max_new_tokens=max_new_tokens, │
│ 166 │ │ │ stopping_criteria=self.stopping_criteria, │
│ │
│ /home/jgzn/anaconda/envs/XrayGpt/lib/python3.9/site-packages/torch/utils/_co │
│ ntextlib.py:115 in decorate_context │
│ │
│ 112 │ @functools.wraps(func) │
│ 113 │ def decorate_context(*args, **kwargs): │
│ 114 │ │ with ctx_factory(): │
│ ❱ 115 │ │ │ return func(*args, **kwargs) │
│ 116 │ │
│ 117 │ return decorate_context │
│ 118 │
│ │
│ /home/jgzn/anaconda/envs/XrayGpt/lib/python3.9/site-packages/transformers/ge │
│ neration/utils.py:1565 in generate │
│ │
│ 1562 │ │ │ ) │
│ 1563 │ │ │ │
│ 1564 │ │ │ # 13. run sample │
│ ❱ 1565 │ │ │ return self.sample( │
│ 1566 │ │ │ │ input_ids, │
│ 1567 │ │ │ │ logits_processor=logits_processor, │
│ 1568 │ │ │ │ logits_warper=logits_warper, │
│ │
│ /home/jgzn/anaconda/envs/XrayGpt/lib/python3.9/site-packages/transformers/ge │
│ neration/utils.py:2612 in sample │
│ │
│ 2609 │ │ │ model_inputs = self.prepare_inputs_for_generation(input_i │
│ 2610 │ │ │ │
│ 2611 │ │ │ # forward pass to get next token │
│ ❱ 2612 │ │ │ outputs = self( │
│ 2613 │ │ │ │ **model_inputs, │
│ 2614 │ │ │ │ return_dict=True, │
│ 2615 │ │ │ │ output_attentions=output_attentions, │
│ │
│ /home/jgzn/anaconda/envs/XrayGpt/lib/python3.9/site-packages/torch/nn/module │
│ s/module.py:1501 in _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or s │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hoo │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /home/jgzn/anaconda/envs/XrayGpt/lib/python3.9/site-packages/accelerate/hook │
│ s.py:156 in new_forward │
│ │
│ 153 │ │ │ with torch.no_grad(): │
│ 154 │ │ │ │ output = old_forward(*args, **kwargs) │
│ 155 │ │ else: │
│ ❱ 156 │ │ │ output = old_forward(*args, **kwargs) │
│ 157 │ │ return module.hf_hook.post_forward(module, output) │
│ 158 │ │
│ 159 │ module.forward = new_forward │
│ │
│ /home/jgzn/PycharmProjects/Xray/XrayGPT-main/xraygpt/models/modeling_llama.p │
│ y:676 in forward │
│ │
│ 673 │ │ return_dict = return_dict if return_dict is not None else self │
│ 674 │ │ │
│ 675 │ │ # decoder outputs consists of (dec_features, layer_state, dec │
│ ❱ 676 │ │ outputs = self.model( │
│ 677 │ │ │ input_ids=input_ids, │
│ 678 │ │ │ attention_mask=attention_mask, │
│ 679 │ │ │ position_ids=position_ids, │
│ │
│ /home/jgzn/anaconda/envs/XrayGpt/lib/python3.9/site-packages/torch/nn/module │
│ s/module.py:1501 in _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or s │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hoo │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /home/jgzn/anaconda/envs/XrayGpt/lib/python3.9/site-packages/accelerate/hook │
│ s.py:156 in new_forward │
│ │
│ 153 │ │ │ with torch.no_grad(): │
│ 154 │ │ │ │ output = old_forward(*args, **kwargs) │
│ 155 │ │ else: │
│ ❱ 156 │ │ │ output = old_forward(*args, **kwargs) │
│ 157 │ │ return module._hf_hook.post_forward(module, output) │
│ 158 │ │
│ 159 │ module.forward = new_forward │
│ │
│ /home/jgzn/PycharmProjects/Xray/XrayGPT-main/xraygpt/models/modeling_llama.p │
│ y:517 in forward │
│ │
│ 514 │ │ │ ) │
│ 515 │ │ │ position_ids = position_ids.unsqueeze(0).view(-1, seq_leng │
│ 516 │ │ else: │
│ ❱ 517 │ │ │ position_ids = position_ids.view(-1, seq_length).long() │
│ 518 │ │ │
│ 519 │ │ # embed positions │
│ 520 │ │ if attention_mask is None: │
╰──────────────────────────────────────────────────────────────────────────────╯
RuntimeError: shape '[-1, 104]' is invalid for input of size 105
您好,非常感谢您的这份工作!
我想请问一下,对于模型的微调,是全参微调还是使用了lora之类的高效参数微调呢,微调的参数量大概是多少
非常感谢您的回复
Thank you for your wonderful work - XrayGPT. I have some questions about XrayGPT that I have to ask you.
I have downloaded the openi annoatations file from https://mbzuaiac-my.sharepoint.com/:u:/g/personal/omkar_thawakar_mbzuai_ac_ae/EVYGprPyzdhOjFlQ2aNJbykBj49SwTGBYmC1uJ7TMswaVQ?e=qdqS8U . And I've downloaded the openi PNG image file from https://openi.nlm.nih.gov/imgs/collections/NLMCXR_png.tgz .
However I found that the image_id
field in filter_cap.json
does not correspond to the PNG image name, what is the reason for this? How do I deal with this issue?
Or I wonder if I could just get these PNG images that match filter_cap.json
, which would speed up our work.
Nice work!
As the paper said in curating high-quality data, the authors further refine the radiology reports. Can we access this dataset? I think the original MIMIC-CXR report seems inappropriate, especially the finding compared to the previous examination.
Great work on the paper! I am intrigued by one step of the dataset prep desribed in README-DATASET.md.
Can you share the gpt-3.5-turbo
prompts that were used for
Elimination of sentences containing comparisons to the patient's prior medical history.
Removal of de-defined symbols "__" while preserving the original meaning.
This will be tremendously helpful in my learning, thank you!
The user requested pyarrow==12.0.0
datasets 2.12.0 depends on pyarrow>=8.0.0
img2dataset 1.41.0 depends on pyarrow<8 and >=6.0.1
and it is also conflicting with fire==0.5.0 as img2dataset needs fire below 0.5
Thanks for making this innovative work of Xray-based LLM "XrayGPT" public. I followed the provided script to learn the training of a multimodal GPT model.
I am curious about the test.py
script:
Test.yaml
file to run the Testing?test.py
?Kind Regards,
Hello, what specific computer configurations are required for this project?
your demo is not working right now. Can you fix it again
I have tried setting up the environment using both pip and conda and have been unable to do so.
These are the error messages I am getting with each method:
My machine is running Mac OS.
Thanks for the work you're doing!
(xraygpt) PS C:\Users\lenovo\Desktop\XrayGPT-main> python demo.py --cfg-path eval_configs/xraygpt_eval.yaml --gpu-id 0
Initializing Chat
Loading VIT
Traceback (most recent call last):
File "C:\Users\lenovo\Desktop\XrayGPT-main\demo.py", line 60, in
model = model_cls.from_config(model_config).to('cuda:{}'.format(args.gpu_id))
File "C:\Users\lenovo\Desktop\XrayGPT-main\xraygpt\models\mini_gpt4.py", line 358, in from_config
model = cls(
File "C:\Users\lenovo\Desktop\XrayGPT-main\xraygpt\models\mini_gpt4.py", line 71, in init
self.visual_encoder, self.ln_vision = self.init_vision_encoder(
File "C:\Users\lenovo\Desktop\XrayGPT-main\xraygpt\models\blip2.py", line 65, in init_vision_encoder
visual_encoder = create_eva_vit_g(
File "C:\Users\lenovo\Desktop\XrayGPT-main\xraygpt\models\eva_vit.py", line 433, in create_eva_vit_g
state_dict = torch.load(cached_file, map_location="cpu")
File "D:\anaconda3\envs\xraygpt\lib\site-packages\torch\serialization.py", line 797, in load
with _open_zipfile_reader(opened_file) as opened_zipfile:
File "D:\anaconda3\envs\xraygpt\lib\site-packages\torch\serialization.py", line 283, in init
super().init(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory
Dear authors,
Great work and efforts!
Quick question, I didn't find the part of the code where the medclip model is loaded as the visual encoder. Could you point me to the place where you load that?
Thank you so much in advance!
HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/home/omkarthawakar/fahad/MiniGPT-4/Vicuna_Radiology_fp16/'. Use repo_type
argument if needed.
How to resolve it.. Let me know we can connect on gmail
When I run the demo.py following the README,I met the problem as below:
Traceback (most recent call last):
File "E:\XrayGPT-main\demo.py", line 60, in
model = model_cls.from_config(model_config).to('cuda:{}'.format(args.gpu_id))
File "E:\envs\xraygpt\lib\site-packages\torch\nn\modules\module.py", line 1160, in to
return self._apply(convert)
File "E:\envs\xraygpt\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply
module._apply(fn)
File "E:\envs\xraygpt\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply
module._apply(fn)
File "E:\envs\xraygpt\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply
module._apply(fn)
[Previous line repeated 3 more times]
File "E:\envs\xraygpt\lib\site-packages\torch\nn\modules\module.py", line 833, in _apply
param_applied = fn(param)
File "E:\envs\xraygpt\lib\site-packages\torch\nn\modules\module.py", line 1158, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
NotImplementedError: Cannot copy out of meta tensor; no data!
My GPU is RTX2060, I hope to know if the reason I meet the problem is the memory of gpu is not enough.
I would be appreciated if any help is given.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.