ziplab / spt Goto Github PK

Does SPT support the selection of sensitive parameters in LLMS?
Very excited to see your work, I want to migrate your work to LLM,
How can I change the code? Can you give me some advice? Thank you very much.

Traceback (most recent call last):
File "train_spt.py", line 18, in
from model.vision_transformer_timm import VisionTransformerSepQKV
File "/home/lyc/TNTprojectz/KE/SPT/model/init.py", line 2, in
from .swin_transformer import *
ModuleNotFoundError: No module named 'model.swin_transformer'

Hello, thanks for providing the code.
I have some questions about calculating sensitivity, and I appreciate it if you could clarify them for me.

1. What values of alpha and beta should generally be used?
2. in your experience, how many batches should be processed for reliable estimation of sensitivity?
3. In L181 what do the values denote? Are they the number of total tunable parameters to select?
4. Could you explain how the sweep is performed in, and why the value of 80 is chosen in L189?
5. can you explain this condition in L282 in your code? When I run the code it only return results with for 1.0, 0.8 and 0.6, and for smaller values the condition does not satisfy apparently.
6. In L279, can you explain why param count is calculated in this way? What is the division by 1e6 performed?
7. In L191 and L196, why param_num is multiplied by 0.02 and 1e6 respectively?
8. When using LoRA, I assume the additional parameters will be merged into the original params after training is done. Is the code for that available?

Thank you in advance.

Hi, great work! We are unable to achieve the results of the experiment described in the title of the paper. But we can reproduce the results of pre-training the model using other datasets. Can you give us some suggestions for experiments? The following are the experimental results under the parameter quantity of 0.4M. Looking forward to hearing from you.

70.82, 92.42, 71.54, 99.28, 87.22, 55.20, 91.22,Natural.
85.57, 95.96, 85.60, 74.31,Specialized.
81.83, 66.81, 49.36, 78.76, 79.02, 49.38, 27.73, 38.11,Structured.

When I set args.get_sensitivity, I find no such .sh.
I need the .sh for get_sensitivity. Can you provide it?
Thanks very much.