pelochus / ezrknpu Goto Github PK

Thank you for your great job. I met sorry wrong answer. Could you please tell me how to fix this?

I have tried this on Arch Linux with vendor 6.1.(unsure)
And Armbian with vendor 6.1.43

mike@rock-5a:~/qwen-1_8B-rk3588$ rkllm ./qwen-chat-1_8B.rkllm
RKLLM starting, please wait...
rkllm-runtime version: 1.0.1

Query platform error!

mike@rock-5a:~/qwen-1_8B-rk3588$ sudo dmesg | grep -i rknpu
[sudo] password for mike:
[ 11.688653] RKNPU fdab0000.npu: Adding to iommu group 0
[ 11.688837] RKNPU fdab0000.npu: RKNPU: rknpu iommu is enabled, using iommu mode
[ 11.688966] RKNPU fdab0000.npu: Looking up rknpu-supply from device tree
[ 11.689597] RKNPU fdab0000.npu: Looking up mem-supply from device tree
[ 11.690126] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdab0000-0xfdabffff]
[ 11.690158] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdac0000-0xfdacffff]
[ 11.690180] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdad0000-0xfdadffff]
[ 11.691367] [drm] Initialized rknpu 0.9.6 20240322 for fdab0000.npu on minor 1
[ 11.692619] RKNPU fdab0000.npu: Looking up rknpu-supply from device tree
[ 11.695165] RKNPU fdab0000.npu: RKNPU: Failed to get specification_serial_number
[ 11.695193] RKNPU fdab0000.npu: Failed to get specification_serial_number
[ 11.695215] RKNPU fdab0000.npu: Failed to get leakage
[ 11.695276] RKNPU fdab0000.npu: Looking up rknpu-supply from device tree
[ 11.695335] debugfs: Directory 'fdab0000.npu-rknpu' with parent 'vdd_npu_s0' already present!
[ 11.710738] RKNPU fdab0000.npu: pvtm=863
[ 11.718856] RKNPU fdab0000.npu: pvtm-volt-sel=3
[ 11.718988] RKNPU fdab0000.npu: Looking up rknpu-supply from device tree
[ 11.719077] debugfs: Directory 'fdab0000.npu-rknpu' with parent 'vdd_npu_s0' already present!
[ 11.719713] RKNPU fdab0000.npu: Looking up mem-supply from device tree
[ 11.721604] RKNPU fdab0000.npu: avs=0
[ 11.721839] RKNPU fdab0000.npu: l=10000 h=85000 hyst=5000 l_limit=0 h_limit=800000000 h_table=0

What other information might be helpful to solve this?

Amazing work! Any chance of a converted vision model? Perhaps Moondream2 1.8B or LLaVA 7B model?

Here are some different scenarios and their results if there is an issue loading models, as well as a solution at the end:

as regular user (after setting ulimit -n 4096):

av1d@ubuntu:~/ez/qwen-1_8B-rk3588$ rkllm qwen-chat-1_8B.rkllm 
loaded template from prompt.txt
rkllm init start
E RKNN: [07:54:33.648] failed to allocate handle, ret: -1, errno: 23, errstr: Too many open files in system
E RKNN: [07:54:33.648] failed to malloc npu memory, size: 6528, flags: 0x2
E RKNN: [07:54:33.648] failed to allocate handle, ret: -1, errno: 23, errstr: Too many open files in system
E RKNN: [07:54:33.648] failed to malloc npu memory, size: 480, flags: 0xa
E RKNN: [07:54:33.648] failed to allocate handle, ret: -1, errno: 23, errstr: Too many open files in system
E RKNN: [07:54:33.648] failed to malloc npu memory, size: 8832, flags: 0x2
E RKNN: [07:54:33.648] failed to allocate handle, ret: -1, errno: 23, errstr: Too many open files in system
E RKNN: [07:54:33.648] failed to malloc npu memory, size: 960, flags: 0xa
[truncated for brievity]

as sudo with -E to preserve environment (and ulimit -n 4096):

av1d@ubuntu:~/ez/qwen-1_8B-rk3588$ sudo -E rkllm qwen-chat-1_8B.rkllm 
loaded template from prompt.txt
rkllm init start
E RKNN: [07:54:49.293] failed to convert handle(1020) to fd, ret: -1, errno: 24, errstr: Too many open files
Segmentation fault

as root (default ulimit -n 1024):

root@ubuntu:/home/av1d/ez/qwen-1_8B-rk3588# rkllm qwen-chat-1_8B.rkllm 
loaded template from prompt.txt
rkllm init start
E RKNN: [07:56:27.818] failed to convert handle(1020) to fd, ret: -1, errno: 24, errstr: Too many open files
Segmentation fault
root@ubuntu:/home/av1d/ez/qwen-1_8B-rk3588# 

it only works as root, and only after setting ulimit to 4096:

root@ubuntu:/home/av1d/ez/qwen-1_8B-rk3588# ulimit -n 4096
root@ubuntu:/home/av1d/ez/qwen-1_8B-rk3588# rkllm qwen-chat-1_8B.rkllm 
loaded template from prompt.txt
rkllm init start
RKLLM init success!

also doesn't work:
editing /etc/security/limits.conf doesn't work (* soft nofile 16384 and * hard nofile 1048576).

The solution for me was to edit /etc/sysctl.conf, add fs.file-max = 1048576, then run sudo sysctl -p after.
Tested on Ubuntu 22.04.3 LTS. Can now load models as regular user without issue.

For ezrknn-toolkit2, it is necessary to use an exact Python version depending on package that will be installed. Default Python version may vary from OS to OS. Solution proposed:

• Install latest compatible (as of writing, Python 3.11)
• Check version and install corresponding one

See https://github.com/Pelochus/ezrknn-toolkit2/tree/master/rknn-toolkit2/packages and https://github.com/Pelochus/ezrknn-toolkit2/tree/master/rknn-toolkit-lite2/packages

This issue will be used as a thread of me (and anyone that wants to contribute) discovering how to update the driver to 0.9.6 since bigger LLMs (for now 2.7B onwards, but could be 2B onwards) cannot run on driver 0.9.5

For now it is interesting the following:

airockchip/rknn-llm#16