`RuntimeError: Last dim of input tensor must be multiple of 64` when loading CogAgent model about bigdl HOT 4 CLOSED

We are looking into this model.

from bigdl.

@notsyncing Hi, you can avoid the RuntimeError: Last dim of input tensor must be multiple of 64 by using the optimize_model API in this way:

modules_to_not_convert = ['w1','w2','act','ffn_ln','w3','drop']
model = optimize_model(model, modules_to_not_convert=modules_to_not_convert)

However, during further testing we find that HF version of CogAgent relies on the xformers package, which only supports cuda right now, so inference on cpu or xpu is not supported for this model.

from bigdl.

@notsyncing Hi, you can avoid the RuntimeError: Last dim of input tensor must be multiple of 64 by using the optimize_model API in this way:

modules_to_not_convert = ['w1','w2','act','ffn_ln','w3','drop']
model = optimize_model(model, modules_to_not_convert=modules_to_not_convert)

However, during further testing we find that HF version of CogAgent relies on the xformers package, which only supports cuda right now, so inference on cpu or xpu is not supported for this model.

I see. Would you mind considering something like auto skipping unsupported modules?

from bigdl.

Thanks for the suggestion. We will consider adding checks and auto skipping the unsupported modules.

from bigdl.