Can't run Mistral quantized on T4 about text-generation-inference HOT 3 OPEN

this is a very non expert opinion so fair warning -

This was my original code -

'base_model_id = "mistralai/Mistral-7B-v0.1"
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16
)
model = AutoModelForCausalLM.from_pretrained(base_model_id,
quantization_config=bnb_config,
device_map= 'auto'
)'

When I ran this code, I faced the error RuntimeError: I expected all tensors to be on the same device, but I found at least two devices, cuda:6 and cuda:7! (when checking argument for argument mat2 in method wrapper_CUDA_mm).

I then also tried setting a fixed device as the default device using torch.cuda.set_device(2) and {'':torch.cuda.current_device()}

Then I faced your error.

After this, I removed quantisation from the model and saw that it was not causing multitasking problems, and this notebook was working without that quantisation in model code -
https://github.com/brevdev/notebooks/blob/main/mistral-finetune-own-data.ipynb

There might be a problem while sending multiple layers to different devices while they are quantised.

from text-generation-inference.

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

from text-generation-inference.

I believe this issue still deserves some attention?

from text-generation-inference.