gmh14 / robnets Goto Github PK
View Code? Open in Web Editor NEW[CVPR 2020] When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks
Home Page: http://www.mit.edu/~yuzhe/robnets.html
License: MIT License
[CVPR 2020] When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks
Home Page: http://www.mit.edu/~yuzhe/robnets.html
License: MIT License
I am new to the field of adversarial attacks. I noticed that the natural accuracies of ResNet18 and ResNet50 in CIFAR10 are 78.38% and 79.15% respectively, which are much lower than other pre-trained models. So, what is the reason for this? Is it because of adversarial training using PGD? I also found a lack of preprocessing operations, e.g., normalization operations, prior to training the models.
We can use this project with pytorchcount to calculate these parameters? thank you very much!
Hi! I appreciate your good work!
Since the supernet is not release, is there anyway we can replicate your results on correlations between architecture density and their robustness? It would be very useful if you could release some of the specific architectures that you sampled. Even a list of 100 out of the total 1000 genotypes you sampled would be very helpful.
Thanks!
Hi, @gmh14 , thanks for our implementation, I notice that you re-implement the SVHN dataset in the dataset.py, kindly want to know what is the point of rewriting the dataset rather than using the torchvision datasets?
Thanks,
if it possible to share the RobFree's .pt file with me, i want to do some evalution on it. thanks a lot.
Excuse me, I want to know the difference between the training time and the memory consumption of the search network process with and without adversarial training? I want to know how much additional time and memory is needed to obtain robnet. I will be grateful for your answers to my questions。
Excuse me, we fail to adversarially train RobNet family on tiny-imagenet from scratch that the accuracy and adversarial accuracy on validation set don't rise at all. Only the learning rate decay strategy is mentioned in your article. Could you please tell me the whole detailed training configuration on tiny-imagenet?
Hi,
Congratulations on the work. It seems really interesting.
I was hoping to use your SuperNet approach for comparing the robustness of different architectures (paper mentioned that you compared over 1000 network architectures). I see that there is a Train section in the Readme but I am guessing it is not for the SuperNet.
Is it possible for you to let us know if we can access the trained SuperNet from somewhere or else how can I achieve the task mentioned above. Any help would be appreciated
Hi,
I am trying to reproduce the results of RobNets and got a little confused with your SVHN implementation.
It seems like the RobNet used for SVHN has 100 classes.
Also the provided checkpoint has 100 classes I think. I was trying to find details with the datasets in your paper but failed to find anything that describes this discrepancy.
Thanks.
Hello, I want to collect some networks with defensive properties and different structures. Can this code achieve this?
I'm still working on the train part of the Robnets code.But I was wondering if you could give me an answer to the former question that Robnets didn't release the training code for the super net.
Does this affect the extraction of subnetworks?
Hi guys,
I want to use a trained super-net for my experiments. Your RobNet_large_v1_cifar10.pth seems to be an option. However, I am afraid of that you have fine-tuned the super-net w.r.t. the architecture hard-coded in architecture_code.py. I read the README but is still not sure.
Thanks!
How do I do archchtecture search on supernet?
I have been trying to do adversarial training on MobileNetV2 for CIFAR10 using a standard codebase, and my natural/robust accuracies are no where near the reported numbers in Table 7 of the supplementary material.
Could you please share the exact training hyperparameters for the MobileNetV2 model? or the model parameters if that is possible. Thanks.
Hi, I am trying to test the result of your checkpoint "RobNet_free_SVHN.pth.tar". Someone has pointed out your SVHN implementation mistake. Therefore, to use the checkpoint, we directly take the first 10 dimensions of the weights and bias of the fully connected layer. And in order to avoid the difference caused by the attacker's implementation, we directly use your PGD attacker.
However, result we got is quite different than that reported in your article.
Ours: acc clean = 94.09%, acc adv = 60.45%.
Your report: acc adv = 55.59%.
Could you please tell me why?
Is it caused by the mistake mentioned above?
import random
import numpy as np
import torch
import torch.utils.data
from torch import nn
import torch.nn.functional as F
from torchvision import transforms, datasets
import models
import architecture_code
import utils
class AttackPGD(nn.Module):
def __init__(self, model, config):
super(AttackPGD, self).__init__()
self.model = model
self.rand = config['random_start']
self.step_size = config['step_size']
self.epsilon = config['epsilon']
self.num_steps = config['num_steps']
self.attack = config['attack']
def forward(self, inputs, targets):
if not self.attack:
return self.model(inputs), inputs
x = inputs.detach()
if self.rand:
x = x + torch.zeros_like(x).uniform_(-self.epsilon, self.epsilon)
for i in range(self.num_steps):
x.requires_grad_()
with torch.enable_grad():
logits = self.model(x)
loss = F.cross_entropy(logits, targets, size_average=False)
grad = torch.autograd.grad(loss, [x])[0]
x = x.detach() + self.step_size * torch.sign(grad.detach())
x = torch.min(torch.max(x, inputs - self.epsilon), inputs + self.epsilon)
x = torch.clamp(x, 0, 1)
return self.model(x), x
def svhn():
test_transform = transforms.Compose([transforms.ToTensor()])
testset = datasets.SVHN(root="/home/zhaojb17/awnas/data/SVHN/", split='test', download=True, transform=test_transform)
queue = torch.utils.data.DataLoader(testset, batch_size=24, pin_memory=True, num_workers=2)
return queue
def set_seed(seed):
# Set seed for system, numpy and pytorch.
if seed is not None:
np.random.seed(seed)
random.seed(seed)
torch.manual_seed(seed)
set_seed(123)
# use RobNet architecture
net = models.robnet(architecture_code.robnet_free)
net = net.cuda()
# load pre-trained model
utils.load_state('./checkpoint/new_svhn.pt', net)
testloader = svhn()
net.eval()
correct, adv_correct, total, n_batch = 0, 0, 0, len(testloader)
# Set attacker
cfg = { "random_start": True,
"step_size": 2.0/255,
"epsilon": 8.0/255,
"num_steps": 100,
"attack": True
}
attacker = AttackPGD(net, cfg)
for i, data in enumerate(testloader):
images, labels = data
images = images.cuda()
labels = labels.cuda()
outputs = net(images)
_, predicted = torch.max(outputs.data, 1)
correct += (predicted == labels).sum().item()
outputs = attacker(images, labels)[0]
_, predicted = torch.max(outputs.data, 1)
adv_correct += (predicted == labels).sum().item()
total += len(images)
print("{}/{}: {} {}".format(i, n_batch, correct / total, adv_correct / total))
correct = correct / total
adv_correct /= total
I am new to the field of adversarial attacks. I noticed that the natural accuracies of ResNet18 and ResNet50 in CIFAR10 are 78.38% and 79.15% respectively, which are much lower than other pre-trained models. So, what is the reason for this? Is it because of adversarial training using PGD? I also found a lack of preprocessing operations, e.g., normalization operations, prior to training the models.
Hi, in your paper, you say "After training the supernet, we randomly sample 1,000 architectures from the supernet and finetune each of them for 3 epochs", how to do this? Constructing a network according to a randomly generated genotype_list and evaluate this network?
Could you please open source the whole training process? I think that would more valuable than this fixed model. Thank you for your time and consideration.
Hi,
I am trying to find the checkpoint file to reload weights of the RobNet_Free model using the following command:
utils.load_state('./checkpoint/RobNet_free_cifar10.pth.tar', net)
But, I can't locate the checkpoint file anywere on github..
If you have shared the checkpoint file already, would you mind pinpointing me to its location?
Otherwise, do I need to train RobNet_ free from scratch using your training code and reuse weights once the model converges, or
do you plan on releasing the checkpoint files anytime soon?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.