load_in_8bit is not working for some huggingface model about bitsandbytes HOT 9 CLOSED

Hi @sanyalsunny111
Thank you very much for your message
Your initial issue is related to the fact that you did not installed the latest version of transformers. Since the new features of the library has not been released yet, you cannot retrieve these features with pip install transformers. Therefore, you have to manually install the latest version by running:
pip install git+https://github.com/huggingface/transformers.git

However, this model does not support device_map=auto yet. This should be addressed in the PR: huggingface/transformers#18683 therefore available as soon as the fix will be validated.
If you want to use this feature, you can directly download the transformers version that contains the ViLT support. I made an example colab that you can try out here
Let me know if anything else is unclear!

from bitsandbytes.

Hi @sanyalsunny111 !
No worries, I think that you still didn't installed the correct version because you have your previous transformers that you probably did not removed.
Could you try this command? pip install --force git+https://github.com/younesbelkada/transformers.git@eee3986ec37e3050c1ee94a63efb13090602eae5
Thanks!

from bitsandbytes.

Hey @younesbelkada Thank you very much sir. It is working fine.

from bitsandbytes.

Great ! Very happy that you made it work! 💪 Do not hesitate to open an issue if you face into any new issue

from bitsandbytes.

@younesbelkada Thank you for your previous response. you rightly mentioned device map auto is not supported yet and without that we cannot run a 8 bit model. But my question is how you have used device_map="auto in the colab link you have shared in your previous comment?

from bitsandbytes.

Hi @sanyalsunny111
If you follow the same installation guidelines as on the google colab I shared you, you should be able to pass "device_map=auto" without any problems

from bitsandbytes.

Hey @younesbelkada the device_map='auto' is actually affecting the distributed data parallel (DDP). I am using 8 GPUs and trying to run a faster inference. Here is the error I am getting model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa", device_map="auto", load_in_8bit=True)

Could you please suggest how to use load_in_8bit with DDP?

from bitsandbytes.

Hi @sanyalsunny111
Thanks for your message!
Did the error happen also with "load_in_8bit=False"? Could you also share the full script to reproduce the issue?
Thanks

from bitsandbytes.

Hey @younesbelkada Sorry to bother you with more error. Yes, with load_in_8bit=False this error happened code attached screenshot-1.

Now when I am not using load_in_8bit at all no error is happening so, it's safe to assume either device_map or load_in_8bit is causing the error. Here is my piece of code and here is the hugging face tutorial which my code is based upon.

from bitsandbytes.