量化 int4，报错"RuntimeError: CUDA Error: no kernel image is available for execution on the device" about baichuan-13b HOT 7 CLOSED

quantizer.py里面的kernel有问题，可以用chatglm2的代码进行替换。
https://gist.github.com/shesung/3acd80c22a19d3e019553ad7e497a707

from baichuan-13b.

我也遇到这个问题，用官方的方式int4量化，推理有问题

from baichuan-13b.

能不能贴一下你的代码？

from baichuan-13b.

def init_model():
    print("init model ...")
    model = AutoModelForCausalLM.from_pretrained(
        "/data/baichuan/Baichuan-13B-Chat",
        torch_dtype=torch.float16,
        trust_remote_code=True
    )
    model = model.quantize(4).cuda()
    model.generation_config = GenerationConfig.from_pretrained(
        "/data/baichuan/Baichuan-13B-Chat"
    )
    tokenizer = AutoTokenizer.from_pretrained(
        "/data/baichuan/Baichuan-13B-Chat",
    #    use_fast=False,
        trust_remote_code=True
    )
    return model, tokenizer

from baichuan-13b.

def init_model():
    print("init model ...")
    model = AutoModelForCausalLM.from_pretrained(
        "/data/baichuan/Baichuan-13B-Chat",
        torch_dtype=torch.float16,
        trust_remote_code=True
    )
    model = model.quantize(4).cuda()
    model.generation_config = GenerationConfig.from_pretrained(
        "/data/baichuan/Baichuan-13B-Chat"
    )
    tokenizer = AutoTokenizer.from_pretrained(
        "/data/baichuan/Baichuan-13B-Chat",
    #    use_fast=False,
        trust_remote_code=True
    )
    return model, tokenizer

我的代码和你差不多：
def init_model():
model = AutoModelForCausalLM.from_pretrained(
"baichuan-inc/Baichuan-13B-Chat",
torch_dtype=torch.float16,
# device_map="auto",
trust_remote_code=True
)
model = model.quantize(4).cuda()
model.generation_config = GenerationConfig.from_pretrained(
"baichuan-inc/Baichuan-13B-Chat"
)
tokenizer = AutoTokenizer.from_pretrained(
"baichuan-inc/Baichuan-13B-Chat",
use_fast=False,
trust_remote_code=True
)
return model, tokenizer

可以正常运行

from baichuan-13b.

def init_model():
    print("init model ...")
    model = AutoModelForCausalLM.from_pretrained(
        "/data/baichuan/Baichuan-13B-Chat",
        torch_dtype=torch.float16,
        trust_remote_code=True
    )
    model = model.quantize(4).cuda()
    model.generation_config = GenerationConfig.from_pretrained(
        "/data/baichuan/Baichuan-13B-Chat"
    )
    tokenizer = AutoTokenizer.from_pretrained(
        "/data/baichuan/Baichuan-13B-Chat",
    #    use_fast=False,
        trust_remote_code=True
    )
    return model, tokenizer

我的代码和你差不多： def init_model(): model = AutoModelForCausalLM.from_pretrained( "baichuan-inc/Baichuan-13B-Chat", torch_dtype=torch.float16, # device_map="auto", trust_remote_code=True ) model = model.quantize(4).cuda() model.generation_config = GenerationConfig.from_pretrained( "baichuan-inc/Baichuan-13B-Chat" ) tokenizer = AutoTokenizer.from_pretrained( "baichuan-inc/Baichuan-13B-Chat", use_fast=False, trust_remote_code=True ) return model, tokenizer

可以正常运行

我的和你一样 无法运行，报错跟楼主一样。

from baichuan-13b.

解决了吗 遇到一样的问题

from baichuan-13b.