pengxiao-song / lawgpt Goto Github PK

请问当前的包只能部署在Linux环境下吗？Windows环境会找不到cuda环境
总会出这方面的报错RuntimeError:
CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

但是我本地包括我的依赖环境确实已经装了cuda。

Hi everyone, I just implemented data storing to the pinecone using OpenAI Embeddings but I can't get data id.
How can I do that?
I used langchain but it's impossible to do that?

在第三步：

3. 运行合并脚本
最后，合并原版 Chinese-LLaMA-7B 模型权重和二次训练到的 legal-lora-7b 权重：

sh scripts/merge.sh

generate.sh 是merge.sh这个文件吗？

参数没修改过，用的是默认的参数。
输入是：酒驾撞人要判多久
结果如图

安装过程参考的readme文件，请问是否遇到过相同的问题，可能是哪里的问题，怎么排查呢，非常感谢~

你好，可以提供一下example中使用的prompt吗？想在其他模型上测试一下效果，对比一下

你好。是基于文档的问答形式吗，就是给到GTP结构化的法律知识条文，用chatGPT生成问题和答案？

作者您好，请问下，继续预训练您的算力情况是怎么样的呢，包括内存，显存，多卡等情况，以及花费的时间。

不知道能和newbing一样给出网络链接一样，给出参考的判决书和具体的法律条款，你这是要革法律工作者的命啊，不过这确实是迟早的事。

--load_8bit True 会出来这个错误，
如果False就不会有错误，但是结果是乱码。

Traceback (most recent call last):
File "/home/jovyan/dluo/github/LaWGPT/webui.py", line 211, in
fire.Fire(main)
File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/home/jovyan/dluo/github/LaWGPT/webui.py", line 42, in main
model = LlamaForCausalLM.from_pretrained(
File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2784, in from_pretrained
) = cls._load_pretrained_model(
File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3125, in _load_pretrained_model
new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/transformers/modeling_utils.py", line 717, in _load_state_dict_into_meta_model
set_module_8bit_tensor_to_device(
File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/transformers/utils/bitsandbytes.py", line 78, in set_module_8bit_tensor_to_device
new_value = bnb.nn.Int8Params(new_value, requires_grad=False, has_fp16_weights=has_fp16_weights).to(device)
File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes/nn/modules.py", line 227, in to
return self.cuda(device)
File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes/nn/modules.py", line 191, in cuda
CB, CBt, SCB, SCBt, coo_tensorB = bnb.functional.double_quant(B)
File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes/functional.py", line 1642, in double_quant
row_stats, col_stats, nnz_row_ptr = get_colrow_absmax(
File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes/functional.py", line 1531, in get_colrow_absmax
lib.cget_col_row_stats(ptrA, ptrRowStats, ptrColStats, ptrNnzrows, ct.c_float(threshold), rows, cols)
File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/ctypes/init.py", line 387, in getattr
func = self.getitem(name)
File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/ctypes/init.py", line 392, in getitem
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cget_col_row_stats

如果设置是False就出来的答案莫名其妙：

GPU环境：

运行pip install -r requirements.txt后未报错，再运行webiu.sh时提示Traceback (most recent call last):
File "/Users/xhh/Desktop/LAW/LaWGPT/webui.py", line 4, in
import fire
ModuleNotFoundError: No module named 'fire'；
执行pip install fire后显示Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Requirement already satisfied: fire in /opt/homebrew/lib/python3.11/site-packages (0.5.0)
Requirement already satisfied: six in /opt/homebrew/lib/python3.11/site-packages (from fire) (1.16.0)
Requirement already satisfied: termcolor in /opt/homebrew/lib/python3.11/site-packages (from fire) (2.3.0)，fire已安装；
运行conda list后显示# Name Version Build Channel
bzip2 1.0.8 h620ffc9_4
ca-certificates 2023.01.10 hca03da5_0
libffi 3.4.4 hca03da5_0
ncurses 6.4 h313beb8_0
openssl 1.1.1t h1a28f6b_0
pip 23.0.1 py310hca03da5_0
python 3.10.11 hc0d8a6c_2
readline 8.2 h1a28f6b_0
setuptools 66.0.0 py310hca03da5_0
sqlite 3.41.2 h80987f9_0
tk 8.6.12 hb8d0fd4_0
tzdata 2023c h04d1e81_0
wheel 0.38.4 py310hca03da5_0
xz 5.4.2 h80987f9_0
zlib 1.2.13 h5a0b063_0 ，表明conda无法识别fire，请问如何解决？

运行generate.sh加载模型正常，gradio发送消息报错，请问哪里出了问题吗？bitsandbytes有bug？
File "/home/yjc/LaWGPT/src/generate.py", line 131, in generate_with_callback
model.generate(**kwargs)
File "/home/yjc/anaconda3/lib/python3.9/site-packages/peft/peft_model.py", line 580, in generate
return self.base_model.generate(**kwargs)
File "/home/yjc/anaconda3/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/yjc/anaconda3/lib/python3.9/site-packages/transformers/generation/utils.py", line 1604, in generate
return self.beam_search(
File "/home/yjc/anaconda3/lib/python3.9/site-packages/transformers/generation/utils.py", line 2902, in beam_search
outputs = self(
File "/home/yjc/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/yjc/anaconda3/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/yjc/anaconda3/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 688, in forward
outputs = self.model(
File "/home/yjc/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/yjc/anaconda3/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 578, in forward
layer_outputs = decoder_layer(
File "/home/yjc/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/yjc/anaconda3/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/yjc/anaconda3/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 293, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/home/yjc/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/yjc/anaconda3/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/yjc/anaconda3/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 197, in forward
query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
File "/home/yjc/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/yjc/anaconda3/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/yjc/anaconda3/lib/python3.9/site-packages/peft/tuners/lora.py", line 502, in forward
result = super().forward(x)
File "/home/yjc/anaconda3/lib/python3.9/site-packages/bitsandbytes/nn/modules.py", line 320, in forward
out = bnb.matmul(x, self.weight, bias=self.bias, state=self.state)
File "/home/yjc/anaconda3/lib/python3.9/site-packages/bitsandbytes/autograd/_functions.py", line 500, in matmul
return MatMul8bitLt.apply(A, B, out, bias, state)
File "/home/yjc/anaconda3/lib/python3.9/site-packages/bitsandbytes/autograd/_functions.py", line 380, in forward
outliers = state.CB[:, state.idx.long()].clone()
TypeError: 'NoneType' object is not subscriptable

例子中不同的问题，我看截图中的参数都不一样，好像参数不同，有的结果会不断重复。

您好！麻烦是否可以给出训练使用了多少卡时的估计吗？
另，法律方向应该由专业人士来做评判，现在有没有效果不好的negative example呢？这并没有在repository里找到。
谢谢!

我准备买个台式机来跑

vi scripts/webui.sh
python webui.py \ --load_8bit True \ --base_model '/root/autodl-tmp/LaWGPT/models/base_models/legal_base-7b' \ --lora_weights '/root/autodl-tmp/LaWGPT/models/lora_weights/lawgpt-lora-7b' \ --prompt_template "law_template" \ --server_name "0.0.0.0" \ --share_gradio True \

没有看到经过处理后的data

如题。

楼主有对比过不扩充词表和扩充此表的差异吗？ 是否真的有必要扩充法律词表

git clone http 后安装依赖，提示未成功克隆Hugging Face Transformers库，请问怎么回事

百度飞桨有免费的GPU服务器额度
https://aistudio.baidu.com/

能不能做个飞桨平台的在线试用版本？

--share_gradio Ture \

应该是True

RuntimeError: expected scalar type Half but found Float

finetune.py 脚本根本训练不了啊

I use the "lawgpt-lora-7b" as the lora weights and run the webui.sh. When I run the code on cpu, the response is good. When I run the code on gpu, the result is embarrassing and the same as the result when I remove the code,

model = PeftModel.from_pretrained(
model,
lora_weights,
torch_dtype=torch.float16,
)

which means I cannot load the lora weights on gpu properly.
So how could I properly use the lawgpt-lora-7b weights which is on the meta device on gpu.

二次训练的时候代码里看数据处理是：每条数据的content直接截短，target和input相同，这样不会导致被截断的部分学不到吗？

d2Vza3kzMjc=

我觉得你们做的项目非常棒，敬佩。我们最近也在思考一个项目，跟你们理念几乎完全一样，但我们的LawGPT是基于法语的（不是中文）。请问不知道你们有没有时间可以私聊？感谢。我的微信是snawandeva。

不太熟悉 conda，执行到 conda activate lawgpt 的时候报错：

(base) ➜  LaWGPT git:(main) conda activate lawgpt


EnvironmentNameNotFound: Could not find conda environment: lawgpt
You can list all discoverable environments with `conda info --envs`

在 mac 和 ubuntu 都遇到了同样的错，有没有人遇到过？

司法考试网的那些都是理论知识
要论实战还得是各种奇葩判决

error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "C:\usertemp\AppData\Local\Temp\pip-install-wz75nok1\quantstats_de28e6f2f2214f57a3b180964244d418\setup.py", line 30, in
with io.open(path.join(here, 'requirements.txt'), encoding='utf-8') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\usertemp\AppData\Local\Temp\pip-install-wz75nok1\quantstats_de28e6f2f2214f57a3b180964244d418\requirements.txt'
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.

Traceback (most recent call last):
  File "/Users/ray/Project/LaWGPT/utils/callbacks.py", line 47, in gentask
    ret = self.mfunc(callback=_callback, **self.kwargs)
  File "/Users/ray/Project/LaWGPT/webui.py", line 140, in generate_with_callback
    model.generate(**kwargs)
  File "/Users/ray/.pyenv/versions/3.10.10/lib/python3.10/site-packages/peft/peft_model.py", line 627, in generate
    outputs = self.base_model.generate(**kwargs)
  File "/Users/ray/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/Users/ray/.pyenv/versions/3.10.10/lib/python3.10/site-packages/transformers/generation/utils.py", line 1518, in generate
    return self.greedy_search(
  File "/Users/ray/.pyenv/versions/3.10.10/lib/python3.10/site-packages/transformers/generation/utils.py", line 2332, in greedy_search
    model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
  File "/Users/ray/.pyenv/versions/3.10.10/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 736, in prepare_inputs_for_generation
    position_ids = attention_mask.long().cumsum(-1) - 1
RuntimeError: MPS does not support cumsum op with int64 input
Traceback (most recent call last):
  File "/Users/ray/.pyenv/versions/3.10.10/lib/python3.10/site-packages/gradio/routes.py", line 408, in run_predict
    output = await app.get_blocks().process_api(
  File "/Users/ray/.pyenv/versions/3.10.10/lib/python3.10/site-packages/gradio/blocks.py", line 1315, in process_api
    result = await self.call_function(
  File "/Users/ray/.pyenv/versions/3.10.10/lib/python3.10/site-packages/gradio/blocks.py", line 1059, in call_function
    prediction = await utils.async_iteration(iterator)
  File "/Users/ray/.pyenv/versions/3.10.10/lib/python3.10/site-packages/gradio/utils.py", line 514, in async_iteration
    return await iterator.__anext__()
  File "/Users/ray/.pyenv/versions/3.10.10/lib/python3.10/site-packages/gradio/interface.py", line 632, in fn
    async for output in iterator:
  File "/Users/ray/.pyenv/versions/3.10.10/lib/python3.10/site-packages/gradio/utils.py", line 507, in __anext__
    return await anyio.to_thread.run_sync(
  File "/Users/ray/.pyenv/versions/3.10.10/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/Users/ray/.pyenv/versions/3.10.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/Users/ray/.pyenv/versions/3.10.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/Users/ray/.pyenv/versions/3.10.10/lib/python3.10/site-packages/gradio/utils.py", line 490, in run_sync_iterator_async
    return next(iterator)
  File "/Users/ray/Project/LaWGPT/webui.py", line 156, in evaluate
    print(decoded_output)
UnboundLocalError: local variable 'decoded_output' referenced before assignment

请问是否可以支持 Apple M2 Max 芯片的推理

请问官方有部署到 huggingface 的计划吗？感觉会更加容易让技术小白尝试研究成果

git clone 以后，安装venv, 然后pip安装好了依赖后，执行以下的命令报错：

(venv) ➜ LaWGPT git:(main) sh src/scripts/generate.sh
/Library/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python: can't open file '/Users/majian/Projects/LaWGPT/generate.py': [Errno 2] No such file or directory

这个generate.py 是在哪里呢。

请问一下大佬，Windows10 ，Python3.8 运行报错，具体信息如下：

想问一下预训练的方法是？

Is anybody here who knows about AWS s3 bucket?
I need to implement file storage to S3 bucket but I don't know the way.
How can I do that?
Please let me know

通过启动git下的代码，log提示：已安装的bitsandbytes版本是在没有GPU支持的情况下编译的
是否该项目目前不支持使用GPU加速；
我在huggingface上面体验了在线的lawgpt，响应速度比较慢。

RT，指令数据生成这块能给些详细介绍吗大佬

开发者您好！

请问一下本项目是否有可能通过docker实现在NAS上的自托管，继而有可能实现类似GPT在任何设备上外网访问本项目的可能性？

编程方面纯小白，只是因为有NAS所以经常用docker托管一些方便自己和工作的项目，但也仅此而已了。如果问题比较蠢请见谅

我在colab上运行起来了，但是感觉输出很慢很慢，不知道是不是cuda没有使用好的原因。有兴趣的可以看看我的链接，直接在colab上跑https://colab.research.google.com/drive/1iQjbbIAFV--MOnd8LhahSO6mnrfeveHs#scrollTo=6G0N6U_CE7KN

Hey 我是来自 langchain-serve 的dev！

如果你们有计划使用fastapi或langchain，有把应用部署在云端的需求，可以了解一下我们的repo。

• Exposes APIs from function definitions locally as well as on the cloud.
• Very few lines of code changes, ease of development remains the same as local.
• Supports both REST & Websocket endpoints
• Serverless/autoscaling endpoints with automatic tls certs.
• Real-time streaming, human-in-the-loop support

谢谢

非常棒的工作！ymcui的repository里指出第一阶段：冻结transformer参数，仅训练embedding，在尽量不干扰原模型的情况下适配新增的中文词向量。这样看来这一步应该是加入新的法律词表以后需要做的事情，有在工作计划里吗？