locuslab / convex_adversarial Goto Github PK

Hi,

I'm testing the robustness (by PGD) of the provided models on CIFAR10 with perturbation epsilon=8/255.

Which one of cifar_resnet_2px.pth and cifar_resnet_8px.pth should I use for evaluation? I found that cifar_resnet_2px.pth gives much higher accuracy and robustness against PGD than cifar_resnet_8px.pth.

Looking forward for your reply.

Need to handle 0D tensors and removal of index broadcasting

Non-sequential models are no longer iterable.

When trying to load one of the pre-trained models like this:

robust_model.load_state_dict(torch.load('models_scaled_l2/mnist_large.pth')['state_dict'])

I get this error:

AttributeError: 'list' object has no attribute 'keys'

Where torch.load('models_scaled_l2/mnist_large.pth')['state_dict'] returns a list of length 6 dictionaries. Am I missing something in how to properly load a pre-trained model?

I would like to evaluate the PGD error, so I uncommented the following line in trainer.py

110  _,pgd_err = _pgd(model, Variable(X), Variable(y), epsilon)

Then, I ran the mnist.py, and get the following error

Traceback (most recent call last):
File "examples/mnist.py", line 136, in
err = evaluate_robust(test_loader, model[0], args.epsilon,
File "/data1/home/research/adversarial/convex_temp/convex_adversarial/examples/trainer.py", line 129, in evaluate_robust
_,pgd_err = _pgd(model, Variable(X), Variable(y), epsilon)
File "/data1/home/research/adversarial/convex_temp/convex_adversarial/examples/attacks.py", line 46, in _pgd
loss.backward()
File "/data1/home/anaconda3/envs/torch/lib/python3.8/site-packages/torch/tensor.py", line 198, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/data1/home/anaconda3/envs/torch/lib/python3.8/site-packages/torch/autograd/init.py", line 98, in backward
Variable._execution_engine.run_backward(
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

How can I solve the problem?

The error is thrown in this line:

DualSequential() fails with TypeError: __init__() missing 2 required positional arguments: 'dual_layers' and 'net'

The mnist_epsilon and primal examples make reference to files that aren't available (likely because of the gitignore). Are these files available elsewhere?

Wondering if anyone has built (or has plans to) build support for nn.MaxPool2d?

scaled_models/mnist_large_0_1.pth appears to be for a network with the small architecture --- the dictionary of parameters only contains four layers and the saved file is ~2mb, 10 times smaller than the scaled_models/mnist_large_0_3.pth file

The same code works with the same value of epsilon when it is changed to a Python built-in float instead.

Tested with PyTorch 0.4.0 (CPU).

hi!
Hope your well!
just getting a few errors running the example -

ImportError Traceback (most recent call last)
in ()
1 import sys
2 sys.path.append("../")
----> 3 from convex_adversarial import robust_loss_batch
4
5 torch.manual_seed(1)

ImportError: cannot import name 'robust_loss_batch'

this is in the notebook example.
Best,
Andrew

Hi,

This is really a nice work and many thanks for making this code public!

I got the following error when I was trying to run the 2D.ipynb. Could you please let me know if there is an additional file that is not committed to Github? Thanks again!

ImportError Traceback (most recent call last)
in ()
1 import sys
2 sys.path.append("../")
----> 3 from convex_adversarial import robust_loss_batch
4
5 torch.manual_seed(1)

ImportError: cannot import name robust_loss_batch

Many of the proofs assume that the label y is an one-hot vector. However, in situations such as Virtual Adversarial Training, we would like to find a perturbation that maximizes the KL-divergence between the class distribution for input and the perturbed input. So it could also be seen as a non-one-hot vector for y. In such cases, could we still find an upper-bound of the loss?

I personally think we could indeed encode the KL-divergence as Eq. (4) by choosing an appropriate c. However, I find it hard to prove the equivalent of Theorem 2 under the assumption that y could be non-one-hot. I am wondering if anyone has already proved or disproved this?

Thanks!

Hi Eric,
Could you please provide us with the CIFAR pretrained model? It seems that you have CIFAR training code here but I was not able to find the CIFAR model under models/ directory.
Thanks!

The linked paper reports results on a network with "small" architecture for the CIFAR10 dataset and a perturbation of 2px. This model is not available in the model_scaled directory.

I'm trying to reproduce the MNIST results.
I modified the code such that the "warmup" (epsilon from 0.05 to 0.1) takes 10 epochs
and then I run
python mnist.py --epochs 100
and the test robust error I got is 6.57%.
So I wonder am I exactly replicating the settings in the paper?
Shall I try different random seed?
should I set --scatter_grad and --alpha_grad? which are False by default.

(also, it seems that l1_proj appears in a few places, but actually wasn't in the dual.py)

After changing the padding of conv2d to zero, error occurred below：

File "fashion_mnist.py", line 59, in
args.alpha_grad, args.scatter_grad, l1_proj=args.l1_proj)
File "/home/songcb/convex_adversarial-master/examples/trainer.py", line 22, in train_robust
scatter_grad=scatter_grad)
File "/home/songcb/.local/lib/python3.6/site-packages/convex_adversarial-0.2-py3.6.egg/convex_adversarial/dual.py", line 182, in robust_loss
File "/home/songcb/.local/lib/python3.6/site-packages/convex_adversarial-0.2-py3.6.egg/convex_adversarial/dual.py", line 56, in init
File "/home/songcb/.local/lib/python3.6/site-packages/convex_adversarial-0.2-py3.6.egg/convex_adversarial/affine.py", line 46, in call
File "/home/songcb/.local/lib/python3.6/site-packages/convex_adversarial-0.2-py3.6.egg/convex_adversarial/affine.py", line 66, in forward
File "/home/songcb/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 837, in linear
output = input.matmul(weight.t())
File "/home/songcb/.local/lib/python3.6/site-packages/torch/autograd/variable.py", line 386, in matmul
return torch.matmul(self, other)
File "/home/songcb/.local/lib/python3.6/site-packages/torch/functional.py", line 173, in matmul
return torch.mm(tensor1, tensor2)
RuntimeError: size mismatch at /pytorch/torch/lib/THC/generic/THCTensorMathBlas.cu:243

Hi Eric,
Could you please provide us with the CIFAR pretrained model? It seems that you have CIFAR training code here but I was not able to find the CIFAR model under models/ directory.
Thanks!

``
python fashion_mnist.py
waitGPU: Waiting for the following conditions, checking every 10 seconds.

• utilization <= 20
  waitGPU: Setting GPU to: [0]
  Traceback (most recent call last):
  File "fashion_mnist.py", line 63, in
  args.alpha_grad, args.scatter_grad, l1_proj=args.l1_proj)
  File "/home/songcb/convex_adversarial-master/examples/trainer.py", line 35, in train_robust
  out = model(Variable(X))
  File "/home/songcb/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
  result = self.forward(*input, **kwargs)
  File "/home/songcb/.local/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
  input = module(input)
  File "/home/songcb/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
  result = self.forward(*input, **kwargs)
  File "/home/songcb/.local/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 55, in forward
  return F.linear(input, self.weight, self.bias)
  File "/home/songcb/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 992, in linear
  return torch.addmm(bias, input, weight.t())
  RuntimeError: size mismatch, m1: [50 x 800], m2: [1568 x 100] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:249

I tried running examples/mnist.py unmodified, and ran into the following error.

Traceback (most recent call last):
  File ".../convex_adversarial/examples/mnist.py", line 96, in <module>
    train_robust(train_loader, model, opt, args.epsilon, t, train_log)
  File ".../convex_adversarial/examples/mnist.py", line 27, in train_robust
    Variable(X), Variable(y))
  File ".../convex_adversarial/convex_adversarial/dual.py", line 169, in robust_loss
    dual = DualNetBounds(net, X[i], epsilon)
  File ".../convex_adversarial/convex_adversarial/dual.py", line 85, in __init__
    subset_eye.scatter_(1, self.I[-1].data.nonzero(), d[self.I[-1]].data[:,None])
  File "/usr/local/lib/python3.5/dist-packages/torch/autograd/variable.py", line 74, in __getitem__
    return MaskedSelect.apply(self, key)
  File "/usr/local/lib/python3.5/dist-packages/torch/autograd/_functions/tensor.py", line 462, in forward
    assert not ctx.needs_input_grad[1], "MaskedSelect can't differentiate the mask"
AssertionError: MaskedSelect can't differentiate the mask

I ran into a similar error running examples/2D.ipynb.

This error does not have to do with the convex_adversarial package, but with torch-0.3.0.post4 and torch-0.3.1 (a quick search of the AssertionError shows many users facing similar issues). Rolling back to torch-0.2.0.post2 solved the problem for me (see previous versions).

I'm on the most recent commit of the master branch. I ran examples/mnist.py using the default configurations, and the training appears to be stuck in the first few epochs - the robust loss and regular loss both hover around 2.3, and the robust and regular error are always equal (around 0.9).

I tried a few different configurations, with varying success. By "success," I mean that training decreases the loss to well below 2 in the first epoch. Here are some examples of what worked/what didn't.

These work (usually, these succeed within a few hundred batches of the first epoch):
python examples/mnist.py --batch_size 30 --epsilon 0.01
python examples/mnist.py --batch_size 30 --epsilon 0.03
python examples/mnist.py --batch_size 30 --epsilon 0.05

These don't work:
python examples/mnist.py --batch_size 30 --epsilon 0.09
python examples/mnist.py --batch_size 30 --epsilon 0.1
python examples/mnist.py

Do you know whether something is going wrong with training for the default configuration?

rewrite with https://pytorch.org/docs/stable/tensors.html?highlight=#torch.Tensor.to to make it CPU compatiable

Hi,
I've run the code like the following for epsilon=8/255=0.03137 on CIFAR-10:
python examples/cifar.py --proj 50 --norm_train l1_median --norm_test l1 --starting_epsilon 0.001 --epsilon 0.03137 --schedule_length 20 --epochs 60 --cuda_ids 1

However, I've noticed that, in the case of CIFAR, you've used the normalization with the standard deviation 0.225 in the loader (link to loader file), but epsilon is just added to the normalized input (link). So do I have to normalize the epsilon? like 0.03137/0.225?

FYI, with eps=0.03137 I can get the errors
Robust error 0.519 Error 0.392
which are much better than those reported in the paper
Robust error 0.792 Error 0.722
and similar to the reported results for eps=2/255 (Note that 2/255~0.03137/0.225)
Robust error 0.528 Error 0.389.

~ Sungyoon

Hi,
I've run mnist.py on a single Titan X (Pascal) with the default settings.
However, the speed is much slower(x3) than that reported in the literature (Table 1).
Scaling provable adversarial defenses
My attempt(=0.19*1200=230s/epoch) vs Report(=74s/epoch)

I think the only difference is that I'm using pytorch 1.4.0 and I've changed the code dual_layers.py (not using 'view' but using 'reshape').

It appears that somehow the provided '.pth' model files are not compatible with torch.load() method. With torch 0.4.0 and python 2.7/3.6 the following command results in a 'UnpicklingError: invalid load key' error.
--> pytorch_network.load_state_dict(torch.load('mnist.pth')

where pytorch_network is the model described in the paper. My impression is that pickle files run into issues when saved and loaded on different platforms. Can you please describe which platform (python & torch versions) is used to generate these files?

Simply I change the mnist_model' input channel to 3 and choose the cifar10 dataset, then I get this error:

File "/home/songcb/convex_adversarial-master/examples/trainer.py", line 22, in train_robust
scatter_grad=scatter_grad)
File "/home/songcb/.local/lib/python3.6/site-packages/convex_adversarial-0.2-py3.6.egg/convex_adversarial/dual.py", line 182, in robust_loss
File "/home/songcb/.local/lib/python3.6/site-packages/convex_adversarial-0.2-py3.6.egg/convex_adversarial/dual.py", line 107, in init
TypeError: new() received an invalid combination of arguments - got (Tensor, int), but expected one of:

• (torch.device device)
• (tuple of ints size, torch.device device)
  didn't match because some of the arguments have invalid types: (Tensor, int)
• (torch.Storage storage)
• (Tensor other)
• (object data, torch.device device)
  didn't match because some of the arguments have invalid types: (Tensor, int)

Dear author
I read your paper with great interest, but I have a question.

I would like to use conv_transpose2d layer in the dual network.
you implemented DualConv2d as belows

class DualConv2d(DualLinear): 
    ...
    def forward(self, *xs): 
       ...
        out = conv2d(x, self.layer.weight, 
                       stride=self.layer.stride,
                       padding=self.layer.padding)
      
    def T(self, *xs): 
       ...
        out = conv_transpose2d(x, self.layer.weight, 
                                 stride=self.layer.stride,
                                 padding=self.layer.padding)
      

As following your implementation, I think I can implement DualConvTranspose2d
such that change conv2d to conv_transpose2d in forward, and conv_transpose2d to conv2d in T as belows

class DualConvTranspose2d(DualLinear): 
    ...
    def forward(self, *xs): 
       ...
        out = conv_transpose2d(x, self.layer.weight, 
                       stride=self.layer.stride,
                       padding=self.layer.padding)
      
    def T(self, *xs): 
       ...
        out = conv2d(x, self.layer.weight, 
                                 stride=self.layer.stride,
                                 padding=self.layer.padding)
      

Is my guess correct?

I have defined a network to train the mnist datasets, and it works well when use my own loss, but when considering the robust_loss in convex_adversarial, there is an error

: No module for layer Reshape

following is my code:

# reshape method for Sequential
class Reshape(nn.Module):
    def __init__(self, *args):
        super(Reshape, self).__init__()
        self.shape = args
    def forward(self, x):
        return x.view(self.shape)

# Sequential model
Net = nn.Sequential(
          nn.Conv2d(1,5,5),
          nn.ReLU(),
          nn.Conv2d(5,10,13),
          nn.ReLU(),
		  nn.Conv2d(10,20,5),
		  nn.ReLU(),
		  nn.Conv2d(20,30,5),
		  nn.ReLU(),
		  Reshape(-1, 4 * 4 * 30),
		  nn.Linear(4 * 4 * 30, 500),
		  nn.ReLU(),
		  nn.Linear(500,10)
        )

Dear LocusLab members,

how difficult would it to add support for network shapes that are not a simple chaint? There are a few applications in which they make sense (at least for experimentation). I'm pretty sure that your adversarial learning method would be able to support this, but from a software engineering point of view, adding this feature looks non-trivial as your code structure does not seem to be made for supporting nestings of "nn.Sequential" layers.

As an example, I've uploaded a corresponding Jupyter Notebook to: https://github.com/progirep/convex_adversarial/blob/master/examples/SplitExample.ipynb

The example network splits up into two separate parts, which are then joined again. I've added a "SplitNetwork" class to my fork of your project.

Such network architectures could make sense for applications in which you expect many dependencies between some groups of input variables, but limited interaction between these groups for the overall classification.

I have tested the cifar_small_8px.pth, but only get a 84.1% upper bound.

This was fixed for linear and relu layers in #12, but remains to be done for the other layers (convolutional/batchnorm/projection relu)

Hi, Thanks for OS. Is there support for computing the guaranteed radius of robustness (i.e. solve (17) in the paper)?

I am trying to understand the original code as understanding original paper seems simpler. While trying to optimize Robust Neural Network in 2D.ipynb I get error as:

IndexError                                Traceback (most recent call last)
Input In [7], in <cell line: 19>()
     18 opt = optim.Adam(robust_net.parameters(), lr=1e-3)
     19 for i in range(1000):
---> 20     robust_ce, robust_err = robust_loss(robust_net, epsilon, X, y)
     21     out = robust_net(X)
     22     l2 = nn.CrossEntropyLoss()(out, Variable(y))

File ~/PhD/Compo/from_git/convex_adversarial/examples/../convex_adversarial/dual.py:189, in robust_loss(net, epsilon, X, y, size_average, alpha_grad, scatter_grad)
    187 if X.is_cuda:
    188     c = c.cuda()
--> 189 f = -dual.g(c)
    190 err = (f.data.max(1)[1] != y.data)
    191 if size_average: 

File ~/PhD/Compo/from_git/convex_adversarial/examples/../convex_adversarial/dual.py:162, in DualNetBounds.g(self, c)
    160 print("self.I_neg[i-1].shape: ", self.I_neg[i-1].shape)
    161 print("self.I_neg[i-1].unsqueeze(1).shape: ", self.I_neg[i-1].unsqueeze(1).shape)
--> 162 out[self.I_neg[i-1].unsqueeze(1)] = 0
    163 if not self.I_empty[i-1]:
    164     if self.alpha_grad: 

IndexError: The shape of the mask [12, 1, 100] at index 1 does not match the shape of the indexed tensor [12, 2, 100] at index 1

I added few print statements in dual.py to debug.
I also changed the import statement from:
from convex_adversarial import robust_loss_batch
to
from convex_adversarial import robust_loss_batch
as it was not able to find robust_loss_batch

Thanks in advance for any help.

I'm trying to modify the MNIST example to start robust training after 100 epochs of training on the regular loss. I modify the training objective selection in examples/mnist.py so that it changes to use the robust objective after a number of epochs. However, it then terminates with a CUDA out of memory error... This doesn't happen if I start robust training from the 0th epoch. Is there an easy fix for this? Thanks