I am using mindspore 1.9 , the gpu is a 24g 3090 and both models ran for a long time without results and finally showed the following reported error.

RuntimeError: For 'BatchMatMul', encountered an exception: cuBLAS Error: cublasGemmStridedBatchedEx failed. Possible reasons: the GPU is occupied by other processes. | Error Number: 15 CUBLAS_STATUS_NOT_SUPPORTED: The functionality requested is not supported.

• C++ Call Stack: (For framework developers)

mindspore/ccsrc/plugin/device/gpu/kernel/math/matmul_gpu_kernel.cc:155 LaunchKernel
mindspore/ccsrc/plugin/device/gpu/kernel/math/matmul_gpu_kernel.cc:152 LaunchKernel
when invoke cublas cublasGemmStridedBatchedEx

If this is your first time, please read our contributor guidelines: https://gitee.com/mindspore/mindspore/blob/master/CONTRIBUTING.md

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

跑样例报错

(ms) root@v4ae1f9ac727411f9a25ff403f1244d0-task0-0:/code/minddiffusion/vision/stablediffusionv2# bash scripts/infer.sh
workspace /code/minddiffusion/vision/stablediffusionv2
WORK DIR:/code/minddiffusion/vision/stablediffusionv2
Loading model from models/stablediffusionv2_512.ckpt
LatentDiffusion: Running in eps-prediction mode
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
param not load: (['first_stage_model.encoder.down.3.downsample.conv.weight', 'first_stage_model.encoder.down.3.downsample.conv.bias', 'first_stage_model.decoder.up.0.upsample.conv.weight', 'first_stage_model.decoder.up.0.upsample.conv.bias'], ['cond_stage_model.transformer.transformer_layer.resblocks.23.attn.attn.in_proj.bias', 'cond_stage_model.transformer.transformer_layer.resblocks.23.attn.attn.in_proj.weight', 'cond_stage_model.transformer.transformer_layer.resblocks.23.attn.attn.out_proj.bias', 'cond_stage_model.transformer.transformer_layer.resblocks.23.attn.attn.out_proj.weight', 'cond_stage_model.transformer.transformer_layer.resblocks.23.c_fc.bias', 'cond_stage_model.transformer.transformer_layer.resblocks.23.c_fc.weight', 'cond_stage_model.transformer.transformer_layer.resblocks.23.c_proj.bias', 'cond_stage_model.transformer.transformer_layer.resblocks.23.c_proj.weight', 'cond_stage_model.transformer.transformer_layer.resblocks.23.ln_1.beta', 'cond_stage_model.transformer.transformer_layer.resblocks.23.ln_1.gamma', 'cond_stage_model.transformer.transformer_layer.resblocks.23.ln_2.beta', 'cond_stage_model.transformer.transformer_layer.resblocks.23.ln_2.gamma'])
Data shape for PLMS sampling is (8, 4, 64, 64)
Running PLMS Sampling with 50 timesteps
Traceback (most recent call last):
  File "txt2img.py", line 294, in <module>
    main()
  File "txt2img.py", line 269, in main
    x_T=start_code
  File "/code/minddiffusion/vision/stablediffusionv2/ldm/models/diffusion/plms.py", line 120, in sample
    unconditional_conditioning=unconditional_conditioning,
  File "/code/minddiffusion/vision/stablediffusionv2/ldm/models/diffusion/plms.py", line 167, in plms_sampling
    old_eps=old_eps, t_next=ts_next)
  File "/code/minddiffusion/vision/stablediffusionv2/ldm/models/diffusion/plms.py", line 230, in p_sample_plms
    e_t = get_model_output(x, t)
  File "/code/minddiffusion/vision/stablediffusionv2/ldm/models/diffusion/plms.py", line 195, in get_model_output
    e_t_uncond, e_t = ops.split((self.model.apply_model(x_in, t_in, c_in)), 0, 2)
  File "/root/miniconda3/envs/ms/lib/python3.7/site-packages/mindspore/ops/function/array_func.py", line 5259, in split
    raise ValueError(f"For split, the value of 'split_size_or_sections' must be more than zero, "
ValueError: For split, the value of 'split_size_or_sections' must be more than zero, but got 0.

猜想可能是mindspore版本接口不适配了。mindspore用的是2.0rc1

环境：
Ascend910 eulerosv2r8.aarch64
mindspore-ascend 1.9.0

任务二：个性化文生图任务
执行 bash scripts/run_db_train.sh报错

报错信息：
process id: 21111
Namespace(betas=[0.9, 0.98], callback_size=1, class_word='猫', data_path='/secHome/FFHQ', decay_steps=0, dropout=0.1, end_learning_rate=1e-07, epochs=5, filter_small_size=True, gradient_accumulation_steps=1, image_filter_size=256, image_size=512, init_loss_scale=65536, loss_scale_factor=2, model_config='/home/data/wukong/minddiffusion/vision/wukong-huahua/configs/v1-train-db-chinese.yaml', optim='adamw', output_path='output/α猫', patch_size=32, pretrained_model_file='wukong-huahua-ms.ckpt', pretrained_model_path='models', random_crop=False, reg_data_path='dataset/reg_cat', save_checkpoint_steps=1000, scale_window=1000, seed=3407, start_learning_rate=1e-06, token='α', train_batch_size=1, train_config='/home/data/wukong/minddiffusion/vision/wukong-huahua/configs/train_db_config.json', train_data_path='dataset/train_cat', train_data_repeats=100, use_parallel=False, warmup_steps=100, weight_decay=0.01)
random seed: 3407
Total training images: 5
Total regularization images: 200
The training data is repeated 100 times, and the total number is 505
rank id 0, sample num is 505
LatentDiffusionDB: Running in eps-prediction mode
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
start loading pretrained_ckpt models/wukong-huahua-ms.ckpt
param not load: ['first_stage_model.encoder.down.3.downsample.conv.weight', 'first_stage_model.encoder.down.3.downsample.conv.bias', 'first_stage_model.decoder.up.0.upsample.conv.weight', 'first_stage_model.decoder.up.0.upsample.conv.bias']
end loading ckpt
start_training...
Traceback (most recent call last):
File "run_db_train.py", line 255, in
main(args)
File "run_db_train.py", line 211, in main
model.train(opts.epochs, dataset, callbacks=callback, dataset_sink_mode=False)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 1050, in train
initial_epoch=initial_epoch)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 98, in wrapper
func(self, *args, **kwargs)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 617, in _train
self._train_process(epoch, train_dataset, list_callback, cb_params, initial_epoch, valid_infos)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 908, in _train_process
outputs = self._train_network(*next_element)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/nn/cell.py", line 596, in call
out = self.compile_and_run(*args)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/nn/cell.py", line 985, in compile_and_run
self.compile(*inputs)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/nn/cell.py", line 957, in compile
jit_config_dict=self._jit_config_dict)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/common/api.py", line 1131, in compile
result = self._graph_executor.compile(obj, args_list, phase, self._use_vm_mode())
RuntimeError: Got unexpected keyword argument: c_concat

• The Traceback of Net Construct Code:

The function call stack (See file '/home/data/wukong/minddiffusion/vision/wukong-huahua/rank_0/om/analyze_fail.dat' for more details. Get instructions about analyze_fail.dat at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):

0 In file /usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/nn/wrap/loss_scale.py:336

    loss = self.network(*inputs)
           ^

1 In file /home/data/wukong/minddiffusion/vision/wukong-huahua/ldm/models/diffusion/ddpm.py:481

    loss_train = self.shared_step(train_x, train_c)
                 ^

2 In file /home/data/wukong/minddiffusion/vision/wukong-huahua/ldm/models/diffusion/ddpm.py:477

    loss = self.p_losses(x, c, t)
           ^

3 In file /home/data/wukong/minddiffusion/vision/wukong-huahua/ldm/models/diffusion/ddpm.py:500

    loss_simple = self.get_loss(model_output, target, mean=False).mean([1, 2, 3])
                                ^

• C++ Call Stack: (For framework developers)

mindspore/core/ir/func_graph_extends.cc:139 GenerateKwParams

If this is your first time, please read our contributor guidelines:
https://github.com/mindspore-lab/mindcv/blob/main/CONTRIBUTING.md

Describe the bug/ 问题描述 (Mandatory / 必填)
于BMS服务器启动工程推理功能时，发生报错
报错日志如下：
(minddiffusion) [root@bms-ynaicc-02 stablediffusionv2]# bash scripts/infer.sh
workspace /home/ma-user/workspace/minddiffusion/vision/stablediffusionv2
WORK DIR:/home/ma-user/workspace/minddiffusion/vision/stablediffusionv2
Loading model from models/stablediffusionv2_512.ckpt
LatentDiffusion: Running in eps-prediction mode
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
param not load: (['first_stage_model.encoder.down.3.downsample.conv.weight', 'first_stage_model.encoder.down.3.downsample.conv.bias', 'first_stage_model.decoder.up.0.upsample.conv.weight', 'first_stage_model.decoder.up.0.upsample.conv.bias'], ['cond_stage_model.transformer.transformer_layer.resblocks.23.attn.attn.in_proj.bias', 'cond_stage_model.transformer.transformer_layer.resblocks.23.attn.attn.in_proj.weight', 'cond_stage_model.transformer.transformer_layer.resblocks.23.attn.attn.out_proj.bias', 'cond_stage_model.transformer.transformer_layer.resblocks.23.attn.attn.out_proj.weight', 'cond_stage_model.transformer.transformer_layer.resblocks.23.c_fc.bias', 'cond_stage_model.transformer.transformer_layer.resblocks.23.c_fc.weight', 'cond_stage_model.transformer.transformer_layer.resblocks.23.c_proj.bias', 'cond_stage_model.transformer.transformer_layer.resblocks.23.c_proj.weight', 'cond_stage_model.transformer.transformer_layer.resblocks.23.ln_1.beta', 'cond_stage_model.transformer.transformer_layer.resblocks.23.ln_1.gamma', 'cond_stage_model.transformer.transformer_layer.resblocks.23.ln_2.beta', 'cond_stage_model.transformer.transformer_layer.resblocks.23.ln_2.gamma'])
Traceback (most recent call last):
File "txt2img.py", line 287, in
main()
File "txt2img.py", line 248, in main
uc = model.get_learned_conditioning(batch_size * [""])
File "/home/ma-user/workspace/minddiffusion/vision/stablediffusionv2/ldm/models/diffusion/ddpm.py", line 276, in get_learned_conditioning
c = self.cond_stage_model.encode(c)
File "/home/ma-user/workspace/minddiffusion/vision/stablediffusionv2/ldm/modules/encoders/modules.py", line 36, in encode
outputs = self.transformer(batch_encoding)
File "/home/ma-user/miniconda3/envs/minddiffusion/lib/python3.7/site-packages/mindspore/nn/cell.py", line 620, in call
out = self.compile_and_run(*args, **kwargs)
File "/home/ma-user/miniconda3/envs/minddiffusion/lib/python3.7/site-packages/mindspore/nn/cell.py", line 939, in compile_and_run
self.compile(*args, **kwargs)
File "/home/ma-user/miniconda3/envs/minddiffusion/lib/python3.7/site-packages/mindspore/nn/cell.py", line 917, in compile
jit_config_dict=self._jit_config_dict, *args, **kwargs)
File "/home/ma-user/miniconda3/envs/minddiffusion/lib/python3.7/site-packages/mindspore/common/api.py", line 1388, in compile
result = self._graph_executor.compile(obj, args, kwargs, phase, self._use_vm_mode())
TypeError: For primitive[BatchMatMul], the input type must be same.
name:[w]:Tensor[Float16].
name:[x]:Tensor[Float32].

• The Traceback of Net Construct Code:

The function call stack (See file '/home/ma-user/workspace/minddiffusion/vision/stablediffusionv2/rank_0/om/analyze_fail.ir' for more details. Get instructions about analyze_fail.ir at https://www.mindspore.cn/search?inputValue=analyze_fail.ir):

0 In file /home/ma-user/workspace/minddiffusion/vision/stablediffusionv2/ldm/modules/encoders/text_encoder.py:150

    x = self.transformer_layer(x)
        ^

1 In file /home/ma-user/workspace/minddiffusion/vision/stablediffusionv2/ldm/modules/encoders/text_encoder.py:111

    return self.resblocks(x)
           ^

2 In file /home/ma-user/miniconda3/envs/minddiffusion/lib/python3.7/site-packages/mindspore/nn/layer/container.py:286

    for cell in self.cell_list:

3 In file /home/ma-user/miniconda3/envs/minddiffusion/lib/python3.7/site-packages/mindspore/nn/layer/container.py:287

        input_data = cell(input_data)
                     ^

4 In file /home/ma-user/workspace/minddiffusion/vision/stablediffusionv2/ldm/modules/encoders/text_encoder.py:96

    x = x + self.attn(self.ln_1(x))
            ^

5 In file /home/ma-user/workspace/minddiffusion/vision/stablediffusionv2/ldm/modules/encoders/text_encoder.py:78

    return self.attn(x, x, x, self.attn_mask)
           ^

6 In file /home/ma-user/workspace/minddiffusion/vision/stablediffusionv2/ldm/modules/encoders/text_encoder.py:60

    attn_output = ops.matmul(attn_output_weights, v)  # bs x (HW + 1) x h
                  ^

7 In file /home/ma-user/miniconda3/envs/minddiffusion/lib/python3.7/site-packages/mindspore/ops/function/math_func.py:8444

if not (isinstance(input, Tensor) and isinstance(other, Tensor)):
^

8 In file /home/ma-user/miniconda3/envs/minddiffusion/lib/python3.7/site-packages/mindspore/ops/function/math_func.py:8449

if input_rank == 2 and other_rank == 2:
^

9 In file /home/ma-user/miniconda3/envs/minddiffusion/lib/python3.7/site-packages/mindspore/ops/function/math_func.py:8453

if input_rank == other_rank and input_rank > 2:
^

10 In file /home/ma-user/miniconda3/envs/minddiffusion/lib/python3.7/site-packages/mindspore/ops/function/math_func.py:8455

    return _batch_matmul(input, other)
           ^

• C++ Call Stack: (For framework developers)

mindspore/core/utils/check_convert_utils.cc:912 _CheckTypeSame

(minddiffusion) [root@bms-ynaicc-02 stablediffusionv2]#

• Hardware Environment(Ascend/GPU/CPU) / 硬件环境:
  Ascend 910B，BMS服务器,操作系统openEuler2.8

• Software Environment / 软件环境 (Mandatory / 必填):
  -- mindspore2.0.0
  -- python3.7.5
  -- cann6.0.rc1，
  -- driver 23.0.rc2(C84)

个性化生成里面，infer.sh文件不存在，应该是inpaint.sh文件才对，请修改

If this is your first time, please read our contributor guidelines:
https://github.com/mindspore-lab/mindcv/blob/main/CONTRIBUTING.md

Describe the bug/ 问题描述 (Mandatory / 必填)
wukong-huahua lora微调from tk.graph import freeze_delta 中tk是什么工具包，仓库里没有看到，也不知道在哪里安装

• Hardware Environment(Ascend/GPU/CPU) / 硬件环境:

Please delete the backend not involved / 请删除不涉及的后端:
/device ascend/GPU/CPU/kirin/等其他芯片: Ascend910

• Software Environment / 软件环境 (Mandatory / 必填):
  -- MindSpore version (e.g., 1.7.0.Bxxx) : 1.9
  -- Python version (e.g., Python 3.7.5) :3.9
  -- OS platform and distribution (e.g., Linux Ubuntu 16.04):16.04
  -- GCC/Compiler version (if compiled from source):

• Excute Mode / 执行模式 (Mandatory / 必填)(PyNative/Graph):

Please delete the mode not involved / 请删除不涉及的模式:
/mode pynative
/mode graph

To Reproduce / 重现步骤 (Mandatory / 必填)
Steps to reproduce the behavior:

1. Go to 'wukong-huahua'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior / 预期结果 (Mandatory / 必填)
A clear and concise description of what you expected to happen.

Screenshots/ 日志 / 截图 (Mandatory / 必填)
If applicable, add screenshots to help explain your problem.

Additional context / 备注 (Optional / 选填)
Add any other context about the problem here.

how to print learning rate in callback or other method.

May I ask if the personalized fine-tuning method of the Wukong model can be applied to stable diffusion? I tried it, and it could run, but there was catastrophic forgetting in stable diffusion. How can I solve this problem? If the learning rate is set exactly the same as the Wukong model, it seems that it cannot learn the features of new images, and the fine-tuning effect does not work, but catastrophic forgetting still occurs. If the learning rate is increased, such as 10 times the personalized fine-tuning of the Wukong model, it can learn the features of new images very well, but catastrophic forgetting becomes more severe.

If this is your first time, please read our contributor guidelines:
https://github.com/mindspore-lab/mindcv/blob/main/CONTRIBUTING.md

minddiffusion/vision/stablediffusionv2
按照readme操作安装requirement.txt，bash script/infer.sh
报错

芯片版本

• Software Environment / 软件环境 (Mandatory / 必填):
  
  
  -- OS platform and distribution (e.g., Linux Ubuntu 18.04):
  
  

cd minddiffusion/vision/stablediffusionv2
bash script/infer.sh

Expected behavior / 预期结果 (Mandatory / 必填)
生成图片

Screenshots/ 日志 / 截图 (Mandatory / 必填)

If this is your first time, please read our contributor guidelines:
https://github.com/mindspore-lab/mindcv/blob/main/CONTRIBUTING.md

Describe the bug/ 问题描述 (Mandatory / 必填)
A clear and concise description of what the bug is.
mindspore1.10 stablediffusionv2 测试
bash scripts/infer.sh 推理验证时报错

• Hardware Environment(Ascend/GPU/CPU) / 硬件环境:

Please delete the backend not involved / 请删除不涉及的后端:
/device ascend/GPU/CPU/kirin/等其他芯片

• Software Environment / 软件环境 (Mandatory / 必填):
  -- MindSpore version (e.g., 1.7.0.Bxxx) :
  -- Python version (e.g., Python 3.7.5) :
  -- OS platform and distribution (e.g., Linux Ubuntu 16.04):
  -- GCC/Compiler version (if compiled from source):

• Excute Mode / 执行模式 (Mandatory / 必填)(PyNative/Graph):

Please delete the mode not involved / 请删除不涉及的模式:
/mode pynative
/mode graph

To Reproduce / 重现步骤 (Mandatory / 必填)
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior / 预期结果 (Mandatory / 必填)
A clear and concise description of what you expected to happen.

Screenshots/ 日志 / 截图 (Mandatory / 必填)
If applicable, add screenshots to help explain your problem.

Additional context / 备注 (Optional / 选填)
Add any other context about the problem here.

如题，我做外扩，输入的图像和mask的尺寸，为w=512, h=384，遇到报错，报错位置在c_cat = ops.concat(c_cat, axis=1)。大概是做连接时尺寸不同，希望能支持自由尺寸。感谢！

print params in graph mode:
print("param", self.net.parameters_dict())
but raise error：
RuntimeError: Unsupported expression 'Yield'.

If this is your first time, please read our contributor guidelines:
https://github.com/mindspore-lab/mindcv/blob/main/CONTRIBUTING.md

Describe the bug/ 问题描述 (Mandatory / 必填)
A clear and concise description of what the bug is.

• Hardware Environment(Ascend/GPU/CPU) / 硬件环境:

Please delete the backend not involved / 请删除不涉及的后端:
/device ascend/GPU/CPU/kirin/等其他芯片

• Software Environment / 软件环境 (Mandatory / 必填):
  -- MindSpore version (e.g., 1.7.0.Bxxx) :
  -- Python version (e.g., Python 3.7.5) :
  -- OS platform and distribution (e.g., Linux Ubuntu 16.04):
  -- GCC/Compiler version (if compiled from source):

• Excute Mode / 执行模式 (Mandatory / 必填)(PyNative/Graph):

Please delete the mode not involved / 请删除不涉及的模式:
/mode pynative
/mode graph

To Reproduce / 重现步骤 (Mandatory / 必填)
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior / 预期结果 (Mandatory / 必填)
A clear and concise description of what you expected to happen.

Screenshots/ 日志 / 截图 (Mandatory / 必填)
If applicable, add screenshots to help explain your problem.

Additional context / 备注 (Optional / 选填)
Add any other context about the problem here.