mbzuai-oryx / xraygpt Goto Github PK

I have tried setting up the environment using both pip and conda and have been unable to do so.
These are the error messages I am getting with each method:

1. conda - ResolvePackageNotFound (for about 20 packages)
2. pip - ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/croot/certifi_1671487769961/work/certifi'

My machine is running Mac OS.

Thanks for the work you're doing!

/home/jgzn/anaconda/envs/XrayGpt/lib/python3.9/site-packages/transformers/generation/utils.py:1255: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)
warnings.warn(
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/jgzn/PycharmProjects/Xray/XrayGPT-main/demo1.py:69 in │
│ │
│ 66 │
│ 67 question = "Could you provide a detailed description of the given x-ray │
│ 68 chat.ask(question, chat_state) │
│ ❱ 69 answer, output_token = chat.answer(chat_state, img_list, num_beams=1, │
│ 70 │ │ │ │ │ │ │ │ temperature=1, │
│ 71 │ │ │ │ │ │ │ │ max_new_tokens=300, │
│ 72 │ │ │ │ │ │ │ │ max_length=2000) │
│ │
│ /home/jgzn/PycharmProjects/Xray/XrayGPT-main/xraygpt/conversation/conversati │
│ on.py:163 in answer │
│ │
│ 160 │ │ │
│ 161 │ │ embs = embs[:, begin_idx:] │
│ 162 │ │ │
│ ❱ 163 │ │ outputs = self.model.llama_model.generate( │
│ 164 │ │ │ inputs_embeds=embs, │
│ 165 │ │ │ max_new_tokens=max_new_tokens, │
│ 166 │ │ │ stopping_criteria=self.stopping_criteria, │
│ │
│ /home/jgzn/anaconda/envs/XrayGpt/lib/python3.9/site-packages/torch/utils/_co │
│ ntextlib.py:115 in decorate_context │
│ │
│ 112 │ @functools.wraps(func) │
│ 113 │ def decorate_context(*args, **kwargs): │
│ 114 │ │ with ctx_factory(): │
│ ❱ 115 │ │ │ return func(*args, **kwargs) │
│ 116 │ │
│ 117 │ return decorate_context │
│ 118 │
│ │
│ /home/jgzn/anaconda/envs/XrayGpt/lib/python3.9/site-packages/transformers/ge │
│ neration/utils.py:1565 in generate │
│ │
│ 1562 │ │ │ ) │
│ 1563 │ │ │ │
│ 1564 │ │ │ # 13. run sample │
│ ❱ 1565 │ │ │ return self.sample( │
│ 1566 │ │ │ │ input_ids, │
│ 1567 │ │ │ │ logits_processor=logits_processor, │
│ 1568 │ │ │ │ logits_warper=logits_warper, │
│ │
│ /home/jgzn/anaconda/envs/XrayGpt/lib/python3.9/site-packages/transformers/ge │
│ neration/utils.py:2612 in sample │
│ │
│ 2609 │ │ │ model_inputs = self.prepare_inputs_for_generation(input_i │
│ 2610 │ │ │ │
│ 2611 │ │ │ # forward pass to get next token │
│ ❱ 2612 │ │ │ outputs = self( │
│ 2613 │ │ │ │ **model_inputs, │
│ 2614 │ │ │ │ return_dict=True, │
│ 2615 │ │ │ │ output_attentions=output_attentions, │
│ │
│ /home/jgzn/anaconda/envs/XrayGpt/lib/python3.9/site-packages/torch/nn/module │
│ s/module.py:1501 in _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or s │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hoo │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /home/jgzn/anaconda/envs/XrayGpt/lib/python3.9/site-packages/accelerate/hook │
│ s.py:156 in new_forward │
│ │
│ 153 │ │ │ with torch.no_grad(): │
│ 154 │ │ │ │ output = old_forward(*args, **kwargs) │
│ 155 │ │ else: │
│ ❱ 156 │ │ │ output = old_forward(*args, **kwargs) │
│ 157 │ │ return module.hf_hook.post_forward(module, output) │
│ 158 │ │
│ 159 │ module.forward = new_forward │
│ │
│ /home/jgzn/PycharmProjects/Xray/XrayGPT-main/xraygpt/models/modeling_llama.p │
│ y:676 in forward │
│ │
│ 673 │ │ return_dict = return_dict if return_dict is not None else self │
│ 674 │ │ │
│ 675 │ │ # decoder outputs consists of (dec_features, layer_state, dec │
│ ❱ 676 │ │ outputs = self.model( │
│ 677 │ │ │ input_ids=input_ids, │
│ 678 │ │ │ attention_mask=attention_mask, │
│ 679 │ │ │ position_ids=position_ids, │
│ │
│ /home/jgzn/anaconda/envs/XrayGpt/lib/python3.9/site-packages/torch/nn/module │
│ s/module.py:1501 in _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or s │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hoo │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /home/jgzn/anaconda/envs/XrayGpt/lib/python3.9/site-packages/accelerate/hook │
│ s.py:156 in new_forward │
│ │
│ 153 │ │ │ with torch.no_grad(): │
│ 154 │ │ │ │ output = old_forward(*args, **kwargs) │
│ 155 │ │ else: │
│ ❱ 156 │ │ │ output = old_forward(*args, **kwargs) │
│ 157 │ │ return module._hf_hook.post_forward(module, output) │
│ 158 │ │
│ 159 │ module.forward = new_forward │
│ │
│ /home/jgzn/PycharmProjects/Xray/XrayGPT-main/xraygpt/models/modeling_llama.p │
│ y:517 in forward │
│ │
│ 514 │ │ │ ) │
│ 515 │ │ │ position_ids = position_ids.unsqueeze(0).view(-1, seq_leng │
│ 516 │ │ else: │
│ ❱ 517 │ │ │ position_ids = position_ids.view(-1, seq_length).long() │
│ 518 │ │ │
│ 519 │ │ # embed positions │
│ 520 │ │ if attention_mask is None: │
╰──────────────────────────────────────────────────────────────────────────────╯
RuntimeError: shape '[-1, 104]' is invalid for input of size 105

Great work on the paper! I am intrigued by one step of the dataset prep desribed in README-DATASET.md.

Can you share the gpt-3.5-turbo prompts that were used for

• Elimination of sentences containing comparisons to the patient's prior medical history.
• Removal of de-defined symbols "__" while preserving the original meaning.

This will be tremendously helpful in my learning, thank you!

your demo is not working right now. Can you fix it again

Are you only training the linear projection layer? In any stage are you training Qformer layers?

When I run the demo.py following the README,I met the problem as below:
Traceback (most recent call last):
File "E:\XrayGPT-main\demo.py", line 60, in
model = model_cls.from_config(model_config).to('cuda:{}'.format(args.gpu_id))
File "E:\envs\xraygpt\lib\site-packages\torch\nn\modules\module.py", line 1160, in to
return self._apply(convert)
File "E:\envs\xraygpt\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply
module._apply(fn)
File "E:\envs\xraygpt\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply
module._apply(fn)
File "E:\envs\xraygpt\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply
module._apply(fn)
[Previous line repeated 3 more times]
File "E:\envs\xraygpt\lib\site-packages\torch\nn\modules\module.py", line 833, in _apply
param_applied = fn(param)
File "E:\envs\xraygpt\lib\site-packages\torch\nn\modules\module.py", line 1158, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
NotImplementedError: Cannot copy out of meta tensor; no data!

My GPU is RTX2060, I hope to know if the reason I meet the problem is the memory of gpu is not enough.
I would be appreciated if any help is given.

/root/autodl-tmp/XrayGLM/chatglm-6b tokenizer! Now you can get tokenizer() everywhere

其中ChatGLM-6B中的tokenizer()需要全部的权重吗？

Nice work!
As the paper said in curating high-quality data, the authors further refine the radiology reports. Can we access this dataset? I think the original MIMIC-CXR report seems inappropriate, especially the finding compared to the previous examination.

Dear authors,
I have some questions about the paper content:
(1) what is the MedVicuna and RadVicuna in Table 1? I cannot find them in the paper or on the Internet;
(2) According to Figure 1, it seems only the Linear Transformation Layer is trained in the whole framework, but why you mentioned in the contributions that "The LLM (Vicuna) is fine-turned on medical data"?
(3) In your work, if only the Linear Transformation Layer is trained while the LLM and MedClip are all frozen?

Thank you for your wonderful work - XrayGPT. I have some questions about XrayGPT that I have to ask you.

I have downloaded the openi annoatations file from https://mbzuaiac-my.sharepoint.com/:u:/g/personal/omkar_thawakar_mbzuai_ac_ae/EVYGprPyzdhOjFlQ2aNJbykBj49SwTGBYmC1uJ7TMswaVQ?e=qdqS8U . And I've downloaded the openi PNG image file from https://openi.nlm.nih.gov/imgs/collections/NLMCXR_png.tgz .

However I found that the image_id field in filter_cap.json does not correspond to the PNG image name, what is the reason for this? How do I deal with this issue?

Or I wonder if I could just get these PNG images that match filter_cap.json, which would speed up our work.

Hello, what specific computer configurations are required for this project?

HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/home/omkarthawakar/fahad/MiniGPT-4/Vicuna_Radiology_fp16/'. Use repo_type argument if needed.

How to resolve it.. Let me know we can connect on gmail

(xraygpt) PS C:\Users\lenovo\Desktop\XrayGPT-main> python demo.py --cfg-path eval_configs/xraygpt_eval.yaml --gpu-id 0
Initializing Chat
Loading VIT
Traceback (most recent call last):
File "C:\Users\lenovo\Desktop\XrayGPT-main\demo.py", line 60, in
model = model_cls.from_config(model_config).to('cuda:{}'.format(args.gpu_id))
File "C:\Users\lenovo\Desktop\XrayGPT-main\xraygpt\models\mini_gpt4.py", line 358, in from_config
model = cls(
File "C:\Users\lenovo\Desktop\XrayGPT-main\xraygpt\models\mini_gpt4.py", line 71, in init
self.visual_encoder, self.ln_vision = self.init_vision_encoder(
File "C:\Users\lenovo\Desktop\XrayGPT-main\xraygpt\models\blip2.py", line 65, in init_vision_encoder
visual_encoder = create_eva_vit_g(
File "C:\Users\lenovo\Desktop\XrayGPT-main\xraygpt\models\eva_vit.py", line 433, in create_eva_vit_g
state_dict = torch.load(cached_file, map_location="cpu")
File "D:\anaconda3\envs\xraygpt\lib\site-packages\torch\serialization.py", line 797, in load
with _open_zipfile_reader(opened_file) as opened_zipfile:
File "D:\anaconda3\envs\xraygpt\lib\site-packages\torch\serialization.py", line 283, in init
super().init(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

He kept answering me in English when I asked questions in the make it language. So what do you think about using other languages for training in fintune rather than pre-training the whole model?

Thanks for making this innovative work of Xray-based LLM "XrayGPT" public. I followed the provided script to learn the training of a multimodal GPT model.

I am curious about the test.py script:

1. What would be the Test.yaml file to run the Testing?
2. What would be the Test set to run test.py?
3. Is there any reference data to evaluate the trained XrayGPT model?

Kind Regards,

Hello,

Thank you for your wonderful work and I would like to ask you:

How to get High Quality and Interactive Summary Xray Datasets?

Maybe it can help us with some other work.

Thank you!

The user requested pyarrow==12.0.0
datasets 2.12.0 depends on pyarrow>=8.0.0
img2dataset 1.41.0 depends on pyarrow<8 and >=6.0.1
and it is also conflicting with fire==0.5.0 as img2dataset needs fire below 0.5

Hello, thank you for sharing your data and code, but I don't know if you missed the pre-trained vicuna_weights

Dear authors,

Great work and efforts!
Quick question, I didn't find the part of the code where the medclip model is loaded as the visual encoder. Could you point me to the place where you load that?

Thank you so much in advance!

Hi,
great work! thanks for publishing you code and data.
What exactly do the radiology conversations in Healthcare_Radiology_vicuna.json contain? It still seems to be related to the data from ChatDoctor - did you somehow filter this data or how did you generate the radiology conversation data?

Thanks!