The misc-scripts from alpindale

20B Models Not Working

Using Aphrodite on colab. I adjusted the settings based on the note (set GPU_Memory_Utilization to 0.9 and don't request more than 200 tokens) but I am getting this from SillyTavern:

RuntimeError

Hey, idk what wrong but whenever I try and let the engine run and generate, at the end I get this instead.

RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx

Hey, I've noticed that whenever I try to use GPTQ quantized models, it doesn't work. I've noticed it sees it as AWQ and not GPTQ, I don't know why that is because it worked fine yesterday. I tried to use "TheBloke/Toppy-M-7B-GPTQ", but it gives me this before it shuts down the notebook.

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/endpoints/kobold/api_server.py", line 334, in <module>
    engine = AsyncAphrodite.from_engine_args(engine_args)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 486, in from_engine_args
    engine = cls(parallel_config.worker_use_ray,
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 270, in __init__
    self.engine = self._init_engine(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 306, in _init_engine
    return engine_class(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/aphrodite_engine.py", line 108, in __init__
    self._init_workers(distributed_init_method)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/aphrodite_engine.py", line 144, in _init_workers
    self._run_workers(
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/aphrodite_engine.py", line 747, in _run_workers
    self._run_workers_in_batch(workers, method, *args, **kwargs))
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/aphrodite_engine.py", line 720, in _run_workers_in_batch
    output = executor(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/task_handler/worker.py", line 72, in load_model
    self.model_runner.load_model()
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/task_handler/model_runner.py", line 36, in load_model
    self.model = get_model(self.model_config)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/modeling/loader.py", line 76, in get_model
    quant_config = get_quant_config(model_config.quantization,
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/modeling/hf_downloader.py", line 94, in get_quant_config
    return quant_cls.from_config(hf_quant_config)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/modeling/layers/quantization/awq.py", line 67, in from_config
    zero_point = cls.get_from_keys(config, ["zero_point"])
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/modeling/layers/quantization/base_config.py", line 50, in get_from_keys
    raise ValueError(f"Cannot find any of {keys} in the model's "
ValueError: Cannot find any of ['zero_point'] in the model's quantization config

alpindale / misc-scripts Goto Github PK

misc-scripts's People

Contributors

Stargazers

Watchers

Forkers

misc-scripts's Issues

20B Models Not Working

RuntimeError

GPTQ quantization

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent