Wanted to ask about how to verify performance against samples for defense methods, you didn't quite understand that part. Looking forward to your reply, thanks a million.

I want to train your model, but I couldn't find the corresponding train.py. How can I train this model?

I want the generated attacks to have an image size of 256x256.

For that, I thought to change the param res to 256

python main.py --model_name resnet18 --save_dir $save_dir --images_root $image_root --label_path $label_path --res 256  #224 

An error shows up then

  File "main.py", line 164, in <module>
    adv_image, clean_acc, adv_acc = run_diffusion_attack(tmp_image, label[ind:ind + 1],
  File "main.py", line 70, in run_diffusion_attack
    adv_image, clean_acc, adv_acc = diff_latent_attack.diffattack(diffusion_model, label, controller,
  File "/home/user/.conda/envs/DiffPurification/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/user/DiffAttack/diff_latent_attack.py", line 374, in diffattack
    before_attention_map = aggregate_attention(prompt, controller, 7, ("up", "down"), True, 0, is_cpu=False)
  File "/home/user/DiffAttack/utils.py", line 17, in aggregate_attention
    out = torch.cat(out, dim=0)
RuntimeError: torch.cat(): expected a non-empty list of Tensors

Besides that, I assume that some parameters must be optimized so that the attack is strong on different parameters. Which parameters are important?

Thank you for your excellent work.

We are confused that the cross attention loss (Ltransfer) is calculated when item.shape[1] == num_pixels and res=7 in aggregate_attention() function. Also, why does item need to be divided by cur_step in the get_average_attention() function? It seems that the mid-layer is not considered for computing cross-attention

Looking forward to your reply, thank you very much.

https://github.com/WindVChen/DiffAttack/blob/523c1af0d531c799565f8b258a540fda3c20c278/attentionControl.py#LL24C8-L26C65

Hi, Thanks for sharing your work! Can you please explain what's happening in the above lines of code.
To my understanding, we are passing the 2*[original latents, latents] and corresponding 2*[unconditional embedding, conditional embedding] through the model. In the above lines of code why are we only taking half of the computed attetnion maps? Are we just storing the attention scores when we pass the conditional embedding?

Thank you very much for your open source, I have encountered the following problem with the pre-trained diffusion model from huggingface.

Traceback (most recent call last):
File "/root/code/DiffAttack-main/main.py", line 107, in
ldm_stable = StableDiffusionPipeline.from_pretrained(pretrained_diffusion_path).to('cuda:0')
File "/root/miniconda3/envs/ldm/lib/python3.8/site-packages/diffusers/pipelines/pipeline_utils.py", line 884, in from_pretrained
cached_folder = cls.download(
File "/root/miniconda3/envs/ldm/lib/python3.8/site-packages/diffusers/pipelines/pipeline_utils.py", line 1208, in download
config_file = hf_hub_download(
File "/root/miniconda3/envs/ldm/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 120, in _inner_fn
return fn(*args, **kwargs)
File "/root/miniconda3/envs/ldm/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 1259, in hf_hub_download
raise LocalEntryNotFoundError(
huggingface_hub.utils._errors.LocalEntryNotFoundError: Connection error, and we cannot find the requested files in the disk cache. Please try again or make sure your Internet connection is on.

Thanks for your excellent work.
I found that it took six hours to train just 1,000 images. This is certainly cost intensive. I would like to ask if this is a personal factor for me or for that model, and also would like to ask if that brute requirement of 1000 images? Would it be a smaller amount of data?
I look forward to your reply, thank you very much.

Hi Chen, great work for adversarial attack using diffusion models, I am trying to run your code but getting the following errors:

python main.py --model_name "inception" --save_dir output --images_root demo/images --label_path demo/labels.txt
Traceback (most recent call last):
File "main.py", line 4, in
import diff_latent_attack
File "/home/Neo2/projects/DiffAttack/diff_latent_attack.py", line 8, in
import other_attacks
File "/home/Neo2/projects/DiffAttack/other_attacks.py", line 16, in
from Finegrained_model import model as otherModel
File "/home/Neo2/projects/DiffAttack/Finegrained_model/model.py", line 3, in
from . import LoadModel
File "/home/Neo2/projects/DiffAttack/Finegrained_model/LoadModel.py", line 4, in
import pretrainedmodels
ModuleNotFoundError: No module named 'pretrainedmodels'

I check the code file in LoadModel.py and find the error location:
from torch import nn
import torch
from torchvision import models
import pretrainedmodels
from .config import pretrained_model

can you give me a hint on how to resolve this error so that I can run the attack on the demo image examples? Thanks!

'HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /stabilityai/stable-diffusion-2-base/resolve/main/model_index.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f845e76d6d0>, 'Connection to huggingface.co timed out. (connect timeout=10)'))' thrown while requesting HEAD https://huggingface.co/stabilityai/stable-diffusion-2-base/resolve/main/model_index.json

Your code currently operates one image at a time, is it possible to adjust the batch_size operation? I look forward to hearing from you!

Hello Chen,

Great work there! I am really enjoying it so far!

I only have a small issue. After the adv examples are crafted and the code starts evaluating on transferability on other models, everything runs smooth until it's time for the tf2torch_adv_inception_v3.npy to be evaluated. Then the code crashes and I get the following error :

" *Transfer to tf2torch_adv_inception_v3
Traceback (most recent call last):
File "/users/40241245/work/scripts/DiffAttack/torch_nets/tf2torch_adv_inception_v3.py", line 14, in load_weights
weights_dict = np.load(weight_file, allow_pickle=True).item()
File "/users/40241245/.local/lib/python3.8/site-packages/numpy/lib/npyio.py", line 417, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: 'pretrained_models/tf2torch_adv_inception_v3.npy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main.py", line 195, in
model_transfer(images, adv_images, label, res, save_path=save_dir, args=args)
File "/users/40241245/work/scripts/DiffAttack/other_attacks.py", line 111, in model_transfer
model = model_selection(name)
File "/users/40241245/work/scripts/DiffAttack/other_attacks.py", line 59, in model_selection
model = net.KitModel(model_path)
File "/users/40241245/work/scripts/DiffAttack/torch_nets/tf2torch_adv_inception_v3.py", line 27, in init
_weights_dict = load_weights(weight_file)
File "/users/40241245/work/scripts/DiffAttack/torch_nets/tf2torch_adv_inception_v3.py", line 16, in load_weights
weights_dict = np.load(weight_file, allow_pickle=True, encoding='bytes').item()
File "/users/40241245/.local/lib/python3.8/site-packages/numpy/lib/npyio.py", line 417, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: 'pretrained_models/tf2torch_adv_inception_v3.npy' "

Any ideas or advise on this? Many Thanks!

Thanks for your excellent work. 😃

We wonder the concrete FSGM parameters used in your paper, since we did not find any code for them in this repo.
Conventionally, the FSGM parameters are attack steps steps, total threshold eps and the step size alpha.
Otherwise, the numerical results should be somewhat explained in the README.md to allow verification.

Thanks again for your response.

*Transfer to deit-b
Traceback (most recent call last):
File "/usr/local/miniconda3/envs/students/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/usr/local/miniconda3/envs/students/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/usr/local/miniconda3/envs/students/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/miniconda3/envs/students/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/usr/local/miniconda3/envs/students/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/usr/local/miniconda3/envs/students/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
conn.connect()
File "/usr/local/miniconda3/envs/students/lib/python3.10/site-packages/urllib3/connection.py", line 358, in connect
self.sock = conn = self._new_conn()
File "/usr/local/miniconda3/envs/students/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7eff29f1e2f0>: Failed to establish a new connection: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/miniconda3/envs/students/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/usr/local/miniconda3/envs/students/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/usr/local/miniconda3/envs/students/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /timm/deit_base_patch16_224.fb_in1k/resolve/main/pytorch_model.bin (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7eff29f1e2f0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/tianfengyuan/.local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1232, in hf_hub_download
metadata = get_hf_file_metadata(
File "/home/tianfengyuan/.local/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "/home/tianfengyuan/.local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1599, in get_hf_file_metadata
r = _request_wrapper(
File "/home/tianfengyuan/.local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 417, in _request_wrapper
response = _request_wrapper(
File "/home/tianfengyuan/.local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 452, in _request_wrapper
return http_backoff(
File "/home/tianfengyuan/.local/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 258, in http_backoff
response = session.request(method=method, url=url, **kwargs)
File "/usr/local/miniconda3/envs/students/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/miniconda3/envs/students/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/home/tianfengyuan/.local/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 63, in send
return super().send(request, *args, **kwargs)
File "/usr/local/miniconda3/envs/students/lib/python3.10/site-packages/requests/adapters.py", line 565, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /timm/deit_base_patch16_224.fb_in1k/resolve/main/pytorch_model.bin (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7eff29f1e2f0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), '(Request ID: 95b9f5fa-b3a6-4355-a9a0-acd37a7082d9)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/tianfengyuan/diff_attack/DiffAttack/main.py", line 203, in
model_transfer(images, adv_images, label, res, save_path=save_dir, args=args)
File "/home/tianfengyuan/diff_attack/DiffAttack/other_attacks.py", line 111, in model_transfer
model = model_selection(name)
File "/home/tianfengyuan/diff_attack/DiffAttack/other_attacks.py", line 37, in model_selection
model = timm.create_model(
File "/home/tianfengyuan/.local/lib/python3.10/site-packages/timm/models/_factory.py", line 114, in create_model
model = create_fn(
File "/home/tianfengyuan/.local/lib/python3.10/site-packages/timm/models/deit.py", line 269, in deit_base_patch16_224
model = _create_deit('deit_base_patch16_224', pretrained=pretrained, **dict(model_args, **kwargs))
File "/home/tianfengyuan/.local/lib/python3.10/site-packages/timm/models/deit.py", line 125, in _create_deit
model = build_model_with_cfg(
File "/home/tianfengyuan/.local/lib/python3.10/site-packages/timm/models/_builder.py", line 393, in build_model_with_cfg
load_pretrained(
File "/home/tianfengyuan/.local/lib/python3.10/site-packages/timm/models/_builder.py", line 186, in load_pretrained
state_dict = load_state_dict_from_hf(pretrained_loc)
File "/home/tianfengyuan/.local/lib/python3.10/site-packages/timm/models/_hub.py", line 188, in load_state_dict_from_hf
cached_file = hf_hub_download(hf_model_id, filename=filename, revision=hf_revision)
File "/home/tianfengyuan/.local/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "/home/tianfengyuan/.local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1349, in hf_hub_download
raise LocalEntryNotFoundError(
huggingface_hub.utils._errors.LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

I may not be able to use a VPN when running the code on the server, and I encounter numerous errors every time it reaches the 'Transfer to deit-b' part. How can I resolve this?

Sorry to interrupt, code error Traceback (most recent call last).
File "main.py", line 153, in
adv_image, clean_acc, adv_acc = run_diffusion_attack(tmp_image, label[ind:ind + 1].
File "main.py", line 72, in run_diffusion_attack
adv_image, clean_acc, adv_acc = diff_latent_attack.diffattack(diffusion_model, label, controller.
File "/usr/local/lib/python3.8/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/202221623173/DiffAttack-main/diff_latent_attack.py", line 369, in diffattack
before_attention_map = aggregate_attention(prompt, controller, 7, ("up", "down"), True, 0, is_cpu=False)
File "/202221623173/DiffAttack-main/utils.py", line 13, in aggregate_attention
for item in attention_maps[f"{location}_{'cross' if is_cross else 'self'}"].
KeyError: 'up_cross'

Can I just replace the inception with a pre-trained resnet18?

Would it run on the demo data seamlessly?

The inception model I reproduced couldn't do what you did. We usually have 229✖229 as input for that model, here it is 224✖224. does this have any effect please? Looking forward to your reply.

Hello,

I am interested in using your DiffAttack on 1D sequences with the aim to make them adversarial against 1D Neural Net classifier (for the specific type of sequences ). I Have a few questions , if you can spare some time.

1. Will the DiffAttack work in 1D setting? I noticed that it is set up to be used with images as a principle. Are there any modifications that need to be done beforehand? What do you suggest?

2. The cross attention and self attention are attributes of the Stable diffusion (as I understand it). Given the fact that it might be difficult to find an already trained Stable diffusion on the specific 1D sequences, what do you think it might be a good approach instead? Will it be able to perform the attack in any sort of modification? What are your thoughts on this?

This is more of a discussion, not an issue per se. We could continue this discussion offline if you want. Let me know!

Appreciate your time.