sclbd / backdoorbench Goto Github PK

Thank you for your great work!
I would like to know why the file "cifar10_preactresnet18_bpp_0_1.zip" in OneDrive, which you are offering, contains too much backdoor train data (>num_total_train_datapoisoning_ratio), and what is the role of cross_test_data folder(the amount of cross_test_data equals to num_total_train_datapoisoning_ratio). Which data should I use to train the poisoned model?

First, think about this benchmark, which helps me to reproduce baseline experiment results quickly. However, I have a few concerns about the code in the backdoor_generate_pindex.py. Specifically, we can see that in https://github.com/SCLBD/BackdoorBench/blob/main/utils/backdoor_generate_pindex.py#L59,

non_zero_array = np.random.choice(np.where(targets == tlabel)[0], round(pratio * len(targets)), replace = False)

Seems like the non_zero_array is selected from the label, maybe there should use targets!=tlabels.

Hi!
There may lack a definition of the get_features, i.e.,

features, labels = get_features(args, model_attack, target_layer, data_loader)
NameError: name 'get_features' is not defined

Could you provide some suggestions on this issue?

Thanks a lot.

hi there, i notice that attacks like lc and ssba need files in ./reource, could you please upload the resource folder with google drive or baidu disk? thank you!

Assuming I have developed some new features on top of the library, how do I merge them into the existing code when BackdoorBench is updated?

Hi, I'm trying to use the BackdoorBench folder of backdoored models to benchmark my defense mechanism for publication, but I cannot access the OneDrive. I get a message saying the link has expired and I do not have access.

I really appreciate your job.
Excuse me. I cannot generate poison data in ssba.py("--attack_train_replace_imgs_path" and "--attack_test_replace_imgs_path" should receive two path for poisoned training data and poisoned testing data, respectively)
We've already installed the packages in requirement.txt file.
We generated the toxic data based on the command in the readme, but the execution reported an error at step 3.

I want to ask how should I solve this problem. Thanks in advance!

Hi BackdoorBench Team,

Firstly, I want to express my admiration for your project.Thank you very much for putting in the effort to design and make this !

But I encountered while attempting to implement the TrojanNN attack method based on your code. It appears that a 'clean_model.pth' is required within the 'resource' directory for the operation to proceed successfully. However, after a thorough review of both the project documentation and the README file, I couldn't find any mention or inclusion of this essential file. I'm concerned that I might have overlooked something or perhaps the file is indeed missing from the repository.

Could you please shed some light on this matter? Any guidance you can offer or provision of the missing file would be greatly appreciated and would significantly aid in my understanding and application of your work.

Thank you in advance for your time and assistance !

I want to attack my own datasets filled with videos using badnets or others, so can you tell me how to run the codes or modify the corresponding codes?

Thank you for your excellent work! I would like to know the target class of Label-Consistent on Tiny ImageNet in Google Drive adv_img folder that you are offering.

I see this repo has just added a new defense method CLP - Channel Lipschitzness-based Pruning.

Can you update the Leaderboard to include the results of the CLP and README file to update your API?

Thank you for such comprehensive work.

Hi BackdooorBench Team,

I think I have found a small issue with the backdoor poison index selection method. In the case where a clean label attack is used, the index selection method errors in some cases. I believe the issue is caused by the pratio not being multiplied by the correct length operator. The issue is on line 59 of "backdoor_generate_poison_index.py." I have provided a code snippet below of the issue and how I think it should be fixed. Notice that pratio should be multiplied by len(np.where(targets == tlabel)[0]) rather then len(targets).

non_zero_array = np.random.choice(np.where(targets == tlabel)[0], round(pratio * len(targets)), replace = False)
non_zero_array = np.random.choice(np.where(targets == tlabel)[0], round(pratio * len(np.where(targets == tlabel)[0])), replace = False)

Hi,
It seems that a resource of lc_atyack is missing:
FileNotFoundError: [Errno 2] No such file or directory: '../resource/label-consistent/data/adv_dataset/cifar10_train.npy
Did I forget to download it?
Thanks!

Hello, thank you for the thorough summary you provided. I encountered a problem while executing a command; when i executing

python ./attack/badnet.py --yaml_path ../config/attack/prototype/cifar10.yaml --patch_mask_path ../resource/badnet/trigger_image.png

there might be an infinite loop in the getattr method within bd_dataset_v2.py. I searched online for a solution and found that reducing the number of num_workers could help. I have already adjusted it to 1, but the problem persists. Could you suggest any other possible solutions?
Thanks!

python ./attack/badnet.py --yaml_path ../config/attack/prototype/cifar10.yaml --patch_mask_path ../resource/badnet/trigger_image.png

On page 16 of the BackdoorBench NeurIPS2022 paper, I noticed that NC is introduced as a method that first detects the backdoored model and then mitigates it through pruning based on the searched trigger. However, for the code in this repo, I find that you perform unlearning instead of pruning. Is that a common operation for the implementation of NC?

I really appreciate your job.
I have encountered the error when running all attacks on tiny-imagenet.

INFO:root:Reminder: plz note that if p_num or pratio exceed the number of possible candidate samples
 then only maximum number of samples will be applied
2022-10-24:21:36:35 [INFO    ] [backdoor_generate_pindex.py:35] Reminder: plz note that if p_num or pratio exceed the number of possible candidate samples
 then only maximum number of samples will be applied
INFO:root:Reminder: priority p_num > pratio, and choosing fix number of sample is prefered if possible 
2022-10-24:21:36:35 [INFO    ] [backdoor_generate_pindex.py:36] Reminder: priority p_num > pratio, and choosing fix number of sample is prefered if possible 
INFO:root:poison num:0.0,real pratio:0.0
2022-10-24:21:36:35 [INFO    ] [backdoor_generate_pindex.py:61] poison num:0.0,real pratio:0.0
No poison sample generated !

I want to ask how should I solve this problem. The current pratio is 0.1, Do I need to drease it to 0.01?
Thanks in advance!

Hi BackdoorBench Team,

Firstly, this tool is amazing. Thank you very much for putting in the effort to design and make this.

I have found an issue with the implementation of the ANP method. In both the evaluate_by_threshold and evaluate_by_number methods, the test ASR value and ACC accuracy difference are used to determine what threshold value is used. However, the test ASR value would not be accessible to the defender when performing pruning as the paper does not assume that backdoor data is available. Could you please clarify if this is an issue or if I am missing something.

Thanks :)

Hi！Excuse me. When we execute shell script 'scp_data.sh', we can't download four datasets.Because we can't connect this server 10.20.12.241:22.Please watch this picture.Thank you!

Thanks for your great work! I notice that you mentioned a public leaderboard for evaluating backdoor attacks against different defense methods. However, I can't find the results for tiny imagenet. Will you please update the results for tiny imagenet? Much appreciated!

Hello, thanks for your excellent work. But I meet a question when I train the Imagenet dataset. It was killed during the pre-process bd dataset creation. I think it may be caused by RAM limitation. Can you help me to solve it?

Many thanks for your excellent work! 😁

Now I have a question. When I used the following setting in Blended Attack.

attack_label_trans: all2all
attack_label_shift_amount: 1
num_classses: 10

But I ran into the following error.

Traceback (most recent call last):
  File "./attack/laser_attack.py", line 347, in <module>
    main()
  File "./attack/laser_attack.py", line 212, in main
    adv_train_ds = prepro_cls_DatasetBD(
  File "./backdoorbench/attack/../utils/bd_dataset.py", line 63, in __init__
    assert len(poison_idx) == len(full_dataset_without_transform)
AssertionError

Did I do something wrong? Looking forward to your reply.

V2.0版本中Label Consistent的数据加载存在一些问题。
在LS攻击的毒化数据生成代码craft_adv_dataset.py中，图像和标签被统一保存了下来，并以npz格式存储，然而在Label Consistent中的89行和95行默认读取的文件格式为npy。

craft_adv_dataset.py中将图像和标签统一保存为npz格式：

Thank you for your excellent work. I found that the hyperparameters --lr_finetuning_init and --lr_unlearning_init seem to be not used in the ./defense/abl/abl.py. Is that a bug?

Hi, SCLBD! Thank you for your excellent work! I have a question for the low-frequency attack: How to generate the trigger by myself for the model of a new architecture?

My command is python ./attack/lc_attack.py --yaml_path ../config/attack/lc/cifar10.yaml --dataset cifar10 --dataset_path --save_folder_name xxx --model resnet18
and the result here is

And I checked your git, it does not has that file either... So how should we train the LC attack?

Thank you!

Hi,

I think in the WaNet Attack All2All attack case, the code to calculate targets_bd is wrong.

Reference code from WaNet's GitHub.

if opt.attack_mode == "all2all":
    targets_bd = torch.remainder(targets[:num_bd] + 1, opt.num_classes)

https://github.com/VinAIResearch/Warping-based_Backdoor_Attack-release/blob/94453080f241053ac7c8cc4717da20806ee17e5c/train.py#L82-L83

there is no fp in your defense folder.

Thank you for sharing your work with the community and for all that you do. Your project has made a difference in the lives of many people, and for that, I am truly grateful.
However, I have some concerns regarding the accuracy of the results displayed on the leaderboard. Specifically, I noticed that the success rate of the traditional attack method blended against defense methods such as nad/nc appears to be too high. After attempting to replicate the nad defense against blended attacks, I found that the results were not as good as those reported in the original paper but also not as bad as those presented on the leaderboard.
Can you review the accuracy of the results on the leaderboard to ensure they are reflective of the true performance of the defense methods?
Thank you for your attention to this matter.

Thank you for sharing your work! It's really helpful.
However (excuse me), I can't generate trigger for low frequency attack with resources/lowFrequency/generate_pattern.py.
There's no errors in execution, but the fooling rate in Universal Perturbation algorithm keeps 0 in 50 epochs （it just doesn't change）, which means the perturbation is not optimized and the agorithm is not working.
The picture below shows the fooling rate in epoch 12.

Here's some information:

• I've already installed the packages in requirement.txt file
• I tried to generate trigger on cifar10 dataset, with several clean models you provided in Google drive, but none of them works.
• I used default.yaml and cifar10.yaml (in resources/lowFrequency) to test, only edit necessary parameters(model path or type, for example), left most parameter unchanged.
• I've removed "{iter}" in code below, for it brings errors in execution. The code is about saving the trigger image and I guess this won't influence the algorithnm. I didn't make other change except this.
  

  BackdoorBench/resource/lowFrequency/generate_pattern.py

  Lines 268 to 272 in 04137b7

  	np.save(f'{save_path_prefix}_{iter}.npy', v)
  	v_lossy_image = np.clip(deepcopy(v) * 255 + 255 / 2, 0, 255).squeeze() # since v is [0,1]
  	Image.fromarray(v_lossy_image.astype(np.uint8)).save(f'{save_path_prefix}_{iter}_lossy.jpg')

Your work is the only pytorch-based implementation for lf attack that I can find. The triggers you provided is useful and I'm grateful, but I still want to generate my own triggers.
I want to ask how can I solve this problem. Thanks in advance!

Very nice work. May I confirm that the LC attack is Label-Consistent Backdoor Attacks as shown in this project. In the paper, the LC attack cites the paper Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks.

FileNotFoundError: [Errno 2] No such file or directory: '../resource/ssba/cifar10_ssba_train_b1.npy'

Great work, but I can't find the link of the paper of Ft_Trojan. Could you please provide it

Thanks so much for the v2.0 update - it's a big quality of life improvement!

Could you update the readme to include scripts for running an attack and defense with v2.0 (or update saying that the same script as v1 work).

Edit: nevermind! I just got confused by the formatting!