Comments (6)
Luo-Z13
commented on August 10, 2024
1
@Luo-Z13, can you please check what is the name of the "model_type" in your base model and the saved checkpoint in the config.json file? Please change it to "geochat", if it is "llava". Let me know if that works.
Thank you very much, it works now.
from geochat.
KjAeRsTuIsK
commented on August 10, 2024
Hi @Luo-Z13, thank you for your interest.
You need to change the image size from 336 to 504.
image = processor.preprocess(image,do_resize=True,crop_size ={'height': 504, 'width': 504},size = {'shortest_edge': 504}, return_tensors='pt')['pixel_values'][0]
can you please change this line in train.py file, line 690,691.
I have made the changes in the codebase as well. Let me know if it works now.
from geochat.
Luo-Z13
commented on August 10, 2024
Hi @Luo-Z13, thank you for your interest. You need to change the image size from 336 to 504. image = processor.preprocess(image,do_resize=True,crop_size ={'height': 504, 'width': 504},size = {'shortest_edge': 504}, return_tensors='pt')['pixel_values'][0] can you please change this line in train.py file, line 690,691. I have made the changes in the codebase as well. Let me know if it works now.
Thank you for the response, the previous issue has now been resolved. However, I am encountering OOM when using 4*A100(40 GB), details are as follows:
File "/miniconda-3/envs/geochat/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 216, in forward
File "/miniconda-3/envs/geochat/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 216, in forward
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
File "/miniconda-3/envs/geochat/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
File "/miniconda-3/envs/geochat/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/miniconda-3/envs/geochat/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 216, in forward
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
File "/miniconda-3/envs/geochat/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/miniconda-3/envs/geochat/lib/python3.10/site-packages/peft/tuners/lora.py", line 822, in forward
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
File "/miniconda-3/envs/geochat/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/miniconda-3/envs/geochat/lib/python3.10/site-packages/peft/tuners/lora.py", line 822, in forward
return forward_call(*args, **kwargs)
File "/miniconda-3/envs/geochat/lib/python3.10/site-packages/peft/tuners/lora.py", line 822, in forward
return forward_call(*args, **kwargs)
File "/miniconda-3/envs/geochat/lib/python3.10/site-packages/peft/tuners/lora.py", line 822, in forward
self.lora_B[self.active_adapter](self.lora_B[self.active_adapter](self.lora_B[self.active_adapter](
torch.cudatorch.cudatorch.cuda...OutOfMemoryErrorOutOfMemoryErrorOutOfMemoryError: : : CUDA out of memory. Tried to allocate 1.04 GiB (GPU 3; 39.39 GiB total capacity; 29.67 GiB already allocated; 1.02 GiB free; 36.57 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONFCUDA out of memory. Tried to allocate 1.07 GiB (GPU 1; 39.39 GiB total capacity; 30.12 GiB already allocated; 397.12 MiB free; 37.16 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONFCUDA out of memory. Tried to allocate 1.04 GiB (GPU 2; 39.39 GiB total capacity; 29.76 GiB already allocated; 911.12 MiB free; 36.66 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
self.lora_B[self.active_adapter](
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.06 GiB (GPU 0; 39.39 GiB total capacity; 29.99 GiB already allocated; 719.12 MiB free; 36.89 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
0%| | 0/2413 [00:44<?, ?it/s]
[2024-03-04 22:06:58,942] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 176970
[2024-03-04 22:06:59,647] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 176971
[2024-03-04 22:06:59,665] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 176972
[2024-03-04 22:06:59,681] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 176973
from geochat.
Luo-Z13
commented on August 10, 2024
Scripts merge_lora_weights.py seems to have an issue at the beginning (from llava... ?). After I changed it from
from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path
to
from geochat.model.builder import load_pretrained_model
from geochat.mm_utils import get_model_name_from_path
an error occurred:
Traceback (most recent call last):
File "GeoChat/scripts/merge_lora_weights.py", line 24, in
merge_lora(args)
File "GeoChat/scripts/merge_lora_weights.py", line 10, in merge_lora
tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.model_base, model_name, device_map='cpu')
File "GeoChat/geochat/model/builder.py", line 110, in load_pretrained_model
model = AutoModelForCausalLM.from_pretrained(model_base, torch_dtype=torch.float16, low_cpu_mem_usage=True, device_map="auto")
File "miniconda-3/envs/geochat/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 461, in from_pretrained
config, kwargs = AutoConfig.from_pretrained(
File "miniconda-3/envs/geochat/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 998, in from_pretrained
config_class = CONFIG_MAPPING[config_dict["model_type"]]
File "miniconda-3/envs/geochat/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 710, in getitem
raise KeyError(key)
KeyError: 'llava'
from geochat.
KjAeRsTuIsK
commented on August 10, 2024
@Luo-Z13, can you please check what is the name of the "model_type" in your base model and the saved checkpoint in the config.json file? Please change it to "geochat", if it is "llava". Let me know if that works.
from geochat.
KjAeRsTuIsK
commented on August 10, 2024
Closing this issue for now, please reopen if you find any difficulties.
from geochat.
Related Issues (20)
- How to calculate the metrics [email protected], [email protected], ROUGE and METEOR score in table 7, 8, 9? HOT 6
- get_chunk method in batch_geochat_scene.py seems to be undefined HOT 1
- Minimum memory for the training process
- how to run the lora finetuned model? HOT 5
- metrics about region captioning HOT 1
- training data corrupted HOT 1
- is training necessary ?
- Model for visual grounding
- Calculation of metrics
- Evaluation results about Grounding
- The results of MiniGPT in the paper HOT 2
- when training had an error!
- License for Commercial use
- merge lora
- how to finetune on my custom dataset
- training data corrupt
- Using transformers to use geochat directly
- The error encountered when using ZeRO-2 for training.
- Could you describe the procedure of reproduce the GeoChat?
- Multi images HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from geochat.