But in the code

I think the self.conv_w is the mapping function but it is different from paper

code: BN -> R -> Conv -> BN -> R -> Conv
paper: L -> R -> L

is the better performance by this code than paper format ？

thanks.

Hi, @hszhao, Thanks for this great work.

I have tested the aggregation and subtraction scripts in the folder of /lib/sa/functions/, from my setup as follows:

cuda/10.0.130
cupy-cuda100-7.7.0
python 3.7

I find it takes around 10mins to finish. Here is the log:

$python subtraction_refpad.py
test case passed
567.34s

Is it the same level of time you have taken. Cause I think the size of the input blocks is really small [2, 8, 5, 5], it looks pretty weird to take 10mins to finish.

Is there anything else that need to be clarified about my setup, please let me know.

Hi, I am just wondering is there any support for 3DCNN using proposed pairwise and patchwise attention.
In my case, I have multiple input image and each of them produce a 4D tensor C,D,H,W. I think your implementation only support 3D tensor C,H,W, is that correct ?

Thank you for the repo! I can reproduce the SAN-10 pairwise / patchwise results. However, for the CNN baselines mentioned in this paper (i.e. ResNet-26, ResNet-38, ResNet-50), I wonder how they are trained. Do you simply replace the SAN network by the ResNet in this code base, keep the optimization settings and train? Or use some other repos such as https://github.com/pytorch/examples/tree/master/imagenet? Thanks!

Hello I had a question regarding your code. In your patchwise attention model you are using these cython kernels for aggregation

I was a bit confused on what exactly is it doing can you explain its functionality? If I have for an example
input data: 1x256x1xWH
weights: 1x32x7**2xWH

How does it perform the hadamard product described in equation 4

Also can I generally confirm the following in SAM module and its mapping to equations 4 and 5 in your paper:
1- conv1: phi, conv2:psi, conv3:beta
2- delta is simple concatenation
3- conv_w: gamma

Thanks for your help.

Hi Zhao,
when I test image of which height != width, errors occurred.
lib/sa/functions/aggregation_refpad.py
n, c_x, c_w, in_height, in_width = 2, 8, 4, 24, 44

RuntimeError: Function AggregationRefpadBackward returned an invalid gradient at index 0 - got [2, 8, 28, 44] but expected shape compatible with [2, 8, 24, 44]

Looking forward to your response!
LIANG

Hi, could you kindly provide the implementation details of how to obtain the results demonstrated in Table 10 in your paper, i.e., the part on robustness to adversarial attacks? I have tried several different targeted PGD methods, but still I found my results are quite different from yours.

The test models are released?

model = san.san(sa_type=1, layers=[3, 2, 3, 5, 2], kernels=[3, 7, 7, 7, 7], num_classes=1000).cuda() model.load_state_dict(torch.load('./san15_patchwise/model/model_best.pth'))

RuntimeError: Error(s) in loading state_dict for SAN:
Missing key(s) in state_dict: "conv_in.weight", "bn_in.weight", "bn_in.bias", "bn_in.running_mean", "bn_in.running_var", "conv0.weight", "bn0.weight", "bn0.bias", "bn0.running_mean", "bn0.running_var", "layer0.0.bn1.weight", "layer0.0.bn1.bias", "layer0.0.bn1.running_mean", "layer0.0.bn1.running_var", "layer0.0.sam.conv1.weight", "layer0.0.sam.conv1.bias", "layer0.0.sam.conv2.weight", "layer0.0.sam.conv2.bias", "layer0.0.sam.conv3.weight", "layer0.0.sam.conv3.bias", "layer0.0.sam.conv_w.0.weight", "layer0.0.sam.conv_w.0.bias", "layer0.0.sam.conv_w.0.running_mean", "layer0.0.sam.conv_w.0.running_var", "layer0.0.sam.conv_w.2.weight", "layer0.0.sam.conv_w.3.weight", "layer0.0.sam.conv_w.3.bias", "layer0.0.sam.conv_w.3.running_mean", "layer0.0.sam.conv_w.3.running_var", "layer0.0.sam.conv_w.5.weight", "layer0.0.sam.conv_w.5.bias", "layer0.0.bn2.weight", "layer0.0.bn2.bias", "layer0.0.bn2.running_mean", "layer0.0.bn2.running_var", "layer0.0.conv.weight", "layer0.0.conv.bias", "layer0.1.bn1.weight", "layer0.1.bn1.bias", "layer0.1.bn1.running_mean", "layer0.1.bn1.running_var", "layer0.1.sam.conv1.weight", "layer0.1.sam.conv1.bias", "layer0.1.sam.conv2.weight", "layer0.1.sam.conv2.bias", "layer0.1.sam.conv3.weight", "layer0.1.sam.conv3.bias", "layer0.1.sam.conv_w.0.weight", "layer0.1.sam.conv_w.0.bias", "layer0.1.sam.conv_w.0.running_mean", "layer0.1.sam.conv_w.0.running_var", "layer0.1.sam.conv_w.2.weight", "layer0.1.sam.conv_w.3.weight", "layer0.1.sam.conv_w.3.bias", "layer0.1.sam.conv_w.3.running_mean", "layer0.1.sam.conv_w.3.running_var", "layer0.1.sam.conv_w.5.weight", "layer0.1.sam.conv_w.5.bias", "layer0.1.bn2.weight", "layer0.1.bn2.bias", "layer0.1.bn2.running_mean", "layer0.1.bn2.running_var", "layer0.1.conv.weight", "layer0.1.conv.bias", "layer0.2.bn1.weight", "layer0.2.bn1.bias", "layer0.2.bn1.running_mean", "layer0.2.bn1.running_var", "layer0.2.sam.conv1.weight", "layer0.2.sam.conv1.bias", "layer0.2.sam.conv2.weight", "layer0.2.sam.conv2.bias", "layer0.2.sam.conv3.weight", "layer0.2.sam.conv3.bias", "layer0.2.sam.conv_w.0.weight", "layer0.2.sam.conv_w.0.bias", "layer0.2.sam.conv_w.0.running_mean", "layer0.2.sam.conv_w.0.running_var", "layer0.2.sam.conv_w.2.weight", "layer0.2.sam.conv_w.3.weight", "layer0.2.sam.conv_w.3.bias", "layer0.2.sam.conv_w.3.running_mean", "layer0.2.sam.conv_w.3.running_var", "layer0.2.sam.conv_w.5.weight", "layer0.2.sam.conv_w.5.bias", "layer0.2.bn2.weight", "layer0.2.bn2.bias", "layer0.2.bn2.running_mean", "layer0.2.bn2.running_var", "layer0.2.conv.weight", "layer0.2.conv.bias", "conv1.weight", "bn1.weight", "bn1.bias", "bn1.running_mean", "bn1.running_var", "layer1.0.bn1.weight", "layer1.0.bn1.bias", "layer1.0.bn1.running_mean", "layer1.0.bn1.running_var", "layer1.0.sam.conv1.weight", "layer1.0.sam.conv1.bias", "layer1.0.sam.conv2.weight", "layer1.0.sam.conv2.bias", "layer1.0.sam.conv3.weight", "layer1.0.sam.conv3.bias", "layer1.0.sam.conv_w.0.weight", "layer1.0.sam.conv_w.0.bias", "layer1.0.sam.conv_w.0.running_mean", "layer1.0.sam.conv_w.0.running_var", "layer1.0.sam.conv_w.2.weight", "layer1.0.sam.conv_w.3.weight", "layer1.0.sam.conv_w.3.bias", "layer1.0.sam.conv_w.3.running_mean", "layer1.0.sam.conv_w.3.running_var", "layer1.0.sam.conv_w.5.weight", "layer1.0.sam.conv_w.5.bias", "layer1.0.bn2.weight", "layer1.0.bn2.bias", "layer1.0.bn2.running_mean", "layer1.0.bn2.running_var", "layer1.0.conv.weight", "layer1.0.conv.bias", "layer1.1.bn1.weight", "layer1.1.bn1.bias", "layer1.1.bn1.running_mean", "layer1.1.bn1.running_var", "layer1.1.sam.conv1.weight", "layer1.1.sam.conv1.bias", "layer1.1.sam.conv2.weight", "layer1.1.sam.conv2.bias", "layer1.1.sam.conv3.weight", "layer1.1.sam.conv3.bias", "layer1.1.sam.conv_w.0.weight", "layer1.1.sam.conv_w.0.bias", "layer1.1.sam.conv_w.0.running_mean", "layer1.1.sam.conv_w.0.running_var", "layer1.1.sam.conv_w.2.weight", "layer1.1.sam.conv_w.3.weight", "layer1.1.sam.conv_w.3.bias", "layer1.1.sam.conv_w.3.running_mean", "layer1.1.sam.conv_w.3.running_var", "layer1.1.sam.conv_w.5.weight", "layer1.1.sam.conv_w.5.bias", "layer1.1.bn2.weight", "layer1.1.bn2.bias", "layer1.1.bn2.running_mean", "layer1.1.bn2.running_var", "layer1.1.conv.weight", "layer1.1.conv.bias", "conv2.weight", "bn2.weight", "bn2.bias", "bn2.running_mean", "bn2.running_var", "layer2.0.bn1.weight", "layer2.0.bn1.bias", "layer2.0.bn1.running_mean", "layer2.0.bn1.running_var", "layer2.0.sam.conv1.weight", "layer2.0.sam.conv1.bias", "layer2.0.sam.conv2.weight", "layer2.0.sam.conv2.bias", "layer2.0.sam.conv3.weight", "layer2.0.sam.conv3.bias", "layer2.0.sam.conv_w.0.weight", "layer2.0.sam.conv_w.0.bias", "layer2.0.sam.conv_w.0.running_mean", "layer2.0.sam.conv_w.0.running_var", "layer2.0.sam.conv_w.2.weight", "layer2.0.sam.conv_w.3.weight", "layer2.0.sam.conv_w.3.bias", "layer2.0.sam.conv_w.3.running_mean", "layer2.0.sam.conv_w.3.running_var", "layer2.0.sam.conv_w.5.weight", "layer2.0.sam.conv_w.5.bias", "layer2.0.bn2.weight", "layer2.0.bn2.bias", "layer2.0.bn2.running_mean", "layer2.0.bn2.running_var", "layer2.0.conv.weight", "layer2.0.conv.bias", "layer2.1.bn1.weight", "layer2.1.bn1.bias", "layer2.1.bn1.running_mean", "layer2.1.bn1.running_var", "layer2.1.sam.conv1.weight", "layer2.1.sam.conv1.bias", "layer2.1.sam.conv2.weight", "layer2.1.sam.conv2.bias", "layer2.1.sam.conv3.weight", "layer2.1.sam.conv3.bias", "layer2.1.sam.conv_w.0.weight", "layer2.1.sam.conv_w.0.bias", "layer2.1.sam.conv_w.0.running_mean", "layer2.1.sam.conv_w.0.running_var", "layer2.1.sam.conv_w.2.weight", "layer2.1.sam.conv_w.3.weight", "layer2.1.sam.conv_w.3.bias", "layer2.1.sam.conv_w.3.running_mean", "layer2.1.sam.conv_w.3.running_var", "layer2.1.sam.conv_w.5.weight", "layer2.1.sam.conv_w.5.bias", "layer2.1.bn2.weight", "layer2.1.bn2.bias", "layer2.1.bn2.running_mean", "layer2.1.bn2.running_var", "layer2.1.conv.weight", "layer2.1.conv.bias", "layer2.2.bn1.weight", "layer2.2.bn1.bias", "layer2.2.bn1.running_mean", "layer2.2.bn1.running_var", "layer2.2.sam.conv1.weight", "layer2.2.sam.conv1.bias", "layer2.2.sam.conv2.weight", "layer2.2.sam.conv2.bias", "layer2.2.sam.conv3.weight", "layer2.2.sam.conv3.bias", "layer2.2.sam.conv_w.0.weight", "layer2.2.sam.conv_w.0.bias", "layer2.2.sam.conv_w.0.running_mean", "layer2.2.sam.conv_w.0.running_var", "layer2.2.sam.conv_w.2.weight", "layer2.2.sam.conv_w.3.weight", "layer2.2.sam.conv_w.3.bias", "layer2.2.sam.conv_w.3.running_mean", "layer2.2.sam.conv_w.3.running_var", "layer2.2.sam.conv_w.5.weight", "layer2.2.sam.conv_w.5.bias", "layer2.2.bn2.weight", "layer2.2.bn2.bias", "layer2.2.bn2.running_mean", "layer2.2.bn2.running_var", "layer2.2.conv.weight", "layer2.2.conv.bias", "conv3.weight", "bn3.weight", "bn3.bias", "bn3.running_mean", "bn3.running_var", "layer3.0.bn1.weight", "layer3.0.bn1.bias", "layer3.0.bn1.running_mean", "layer3.0.bn1.running_var", "layer3.0.sam.conv1.weight", "layer3.0.sam.conv1.bias", "layer3.0.sam.conv2.weight", "layer3.0.sam.conv2.bias", "layer3.0.sam.conv3.weight", "layer3.0.sam.conv3.bias", "layer3.0.sam.conv_w.0.weight", "layer3.0.sam.conv_w.0.bias", "layer3.0.sam.conv_w.0.running_mean", "layer3.0.sam.conv_w.0.running_var", "layer3.0.sam.conv_w.2.weight", "layer3.0.sam.conv_w.3.weight", "layer3.0.sam.conv_w.3.bias", "layer3.0.sam.conv_w.3.running_mean", "layer3.0.sam.conv_w.3.running_var", "layer3.0.sam.conv_w.5.weight", "layer3.0.sam.conv_w.5.bias", "layer3.0.bn2.weight", "layer3.0.bn2.bias", "layer3.0.bn2.running_mean", "layer3.0.bn2.running_var", "layer3.0.conv.weight", "layer3.0.conv.bias", "layer3.1.bn1.weight", "layer3.1.bn1.bias", "layer3.1.bn1.running_mean", "layer3.1.bn1.running_var", "layer3.1.sam.conv1.weight", "layer3.1.sam.conv1.bias", "layer3.1.sam.conv2.weight", "layer3.1.sam.conv2.bias", "layer3.1.sam.conv3.weight", "layer3.1.sam.conv3.bias", "layer3.1.sam.conv_w.0.weight", "layer3.1.sam.conv_w.0.bias", "layer3.1.sam.conv_w.0.running_mean", "layer3.1.sam.conv_w.0.running_var", "layer3.1.sam.conv_w.2.weight", "layer3.1.sam.conv_w.3.weight", "layer3.1.sam.conv_w.3.bias", "layer3.1.sam.conv_w.3.running_mean", "layer3.1.sam.conv_w.3.running_var", "layer3.1.sam.conv_w.5.weight", "layer3.1.sam.conv_w.5.bias", "layer3.1.bn2.weight", "layer3.1.bn2.bias", "layer3.1.bn2.running_mean", "layer3.1.bn2.running_var", "layer3.1.conv.weight", "layer3.1.conv.bias", "layer3.2.bn1.weight", "layer3.2.bn1.bias", "layer3.2.bn1.running_mean", "layer3.2.bn1.running_var", "layer3.2.sam.conv1.weight", "layer3.2.sam.conv1.bias", "layer3.2.sam.conv2.weight", "layer3.2.sam.conv2.bias", "layer3.2.sam.conv3.weight", "layer3.2.sam.conv3.bias", "layer3.2.sam.conv_w.0.weight", "layer3.2.sam.conv_w.0.bias", "layer3.2.sam.conv_w.0.running_mean", "layer3.2.sam.conv_w.0.running_var", "layer3.2.sam.conv_w.2.weight", "layer3.2.sam.conv_w.3.weight", "layer3.2.sam.conv_w.3.bias", "layer3.2.sam.conv_w.3.running_mean", "layer3.2.sam.conv_w.3.running_var", "layer3.2.sam.conv_w.5.weight", "layer3.2.sam.conv_w.5.bias", "layer3.2.bn2.weight", "layer3.2.bn2.bias", "layer3.2.bn2.running_mean", "layer3.2.bn2.running_var", "layer3.2.conv.weight", "layer3.2.conv.bias", "layer3.3.bn1.weight", "layer3.3.bn1.bias", "layer3.3.bn1.running_mean", "layer3.3.bn1.running_var", "layer3.3.sam.conv1.weight", "layer3.3.sam.conv1.bias", "layer3.3.sam.conv2.weight", "layer3.3.sam.conv2.bias", "layer3.3.sam.conv3.weight", "layer3.3.sam.conv3.bias", "layer3.3.sam.conv_w.0.weight", "layer3.3.sam.conv_w.0.bias", "layer3.3.sam.conv_w.0.running_mean", "layer3.3.sam.conv_w.0.running_var", "layer3.3.sam.conv_w.2.weight", "layer3.3.sam.conv_w.3.weight", "layer3.3.sam.conv_w.3.bias", "layer3.3.sam.conv_w.3.running_mean", "layer3.3.sam.conv_w.3.running_var", "layer3.3.sam.conv_w.5.weight", "layer3.3.sam.conv_w.5.bias", "layer3.3.bn2.weight", "layer3.3.bn2.bias", "layer3.3.bn2.running_mean", "layer3.3.bn2.running_var", "layer3.3.conv.weight", "layer3.3.conv.bias", "layer3.4.bn1.weight", "layer3.4.bn1.bias", "layer3.4.bn1.running_mean", "layer3.4.bn1.running_var", "layer3.4.sam.conv1.weight", "layer3.4.sam.conv1.bias", "layer3.4.sam.conv2.weight", "layer3.4.sam.conv2.bias", "layer3.4.sam.conv3.weight", "layer3.4.sam.conv3.bias", "layer3.4.sam.conv_w.0.weight", "layer3.4.sam.conv_w.0.bias", "layer3.4.sam.conv_w.0.running_mean", "layer3.4.sam.conv_w.0.running_var", "layer3.4.sam.conv_w.2.weight", "layer3.4.sam.conv_w.3.weight", "layer3.4.sam.conv_w.3.bias", "layer3.4.sam.conv_w.3.running_mean", "layer3.4.sam.conv_w.3.running_var", "layer3.4.sam.conv_w.5.weight", "layer3.4.sam.conv_w.5.bias", "layer3.4.bn2.weight", "layer3.4.bn2.bias", "layer3.4.bn2.running_mean", "layer3.4.bn2.running_var", "layer3.4.conv.weight", "layer3.4.conv.bias", "conv4.weight", "bn4.weight", "bn4.bias", "bn4.running_mean", "bn4.running_var", "layer4.0.bn1.weight", "layer4.0.bn1.bias", "layer4.0.bn1.running_mean", "layer4.0.bn1.running_var", "layer4.0.sam.conv1.weight", "layer4.0.sam.conv1.bias", "layer4.0.sam.conv2.weight", "layer4.0.sam.conv2.bias", "layer4.0.sam.conv3.weight", "layer4.0.sam.conv3.bias", "layer4.0.sam.conv_w.0.weight", "layer4.0.sam.conv_w.0.bias", "layer4.0.sam.conv_w.0.running_mean", "layer4.0.sam.conv_w.0.running_var", "layer4.0.sam.conv_w.2.weight", "layer4.0.sam.conv_w.3.weight", "layer4.0.sam.conv_w.3.bias", "layer4.0.sam.conv_w.3.running_mean", "layer4.0.sam.conv_w.3.running_var", "layer4.0.sam.conv_w.5.weight", "layer4.0.sam.conv_w.5.bias", "layer4.0.bn2.weight", "layer4.0.bn2.bias", "layer4.0.bn2.running_mean", "layer4.0.bn2.running_var", "layer4.0.conv.weight", "layer4.0.conv.bias", "layer4.1.bn1.weight", "layer4.1.bn1.bias", "layer4.1.bn1.running_mean", "layer4.1.bn1.running_var", "layer4.1.sam.conv1.weight", "layer4.1.sam.conv1.bias", "layer4.1.sam.conv2.weight", "layer4.1.sam.conv2.bias", "layer4.1.sam.conv3.weight", "layer4.1.sam.conv3.bias", "layer4.1.sam.conv_w.0.weight", "layer4.1.sam.conv_w.0.bias", "layer4.1.sam.conv_w.0.running_mean", "layer4.1.sam.conv_w.0.running_var", "layer4.1.sam.conv_w.2.weight", "layer4.1.sam.conv_w.3.weight", "layer4.1.sam.conv_w.3.bias", "layer4.1.sam.conv_w.3.running_mean", "layer4.1.sam.conv_w.3.running_var", "layer4.1.sam.conv_w.5.weight", "layer4.1.sam.conv_w.5.bias", "layer4.1.bn2.weight", "layer4.1.bn2.bias", "layer4.1.bn2.running_mean", "layer4.1.bn2.running_var", "layer4.1.conv.weight", "layer4.1.conv.bias", "fc.weight", "fc.bias".
Unexpected key(s) in state_dict: "epoch", "state_dict", "optimizer", "scheduler", "top1_val", "top5_val".

Hello Dr. Zhao, I was studying your paper and the code shared on the page but had difficulty understanding the "load_kernel()" and "f(block=...., grid=(...., args=[..., stream=...)" parts in subtraction/subtraction2/aggregate functions. How do they affect the subtraction relation function and aggregation function and will the results differ without them?

Any help would be appreciated. Thank you

Dear Dr.Zhao,
I have a question. Have you tried to apply the model to the task of segmentation and detection?

Thank you for your great work!

When I run the SAM module on PatchWise-Attention (sa_type=1) for verification, the input tensors of the Aggregation function has the following shape.

In the paper, it says that the outputs of the streams are aggregated via a Hadamard product.

Would you like to tell me what operations are performed on the tensors of these different shapes to perform a Hadamard product?

Hi, @hszhao ,

Thanks for releasing the package. I'd rather to know the minimum requirement on GPU memory. Is a GPU with 12GB enough to run the training?

THX!

enviroment:
pytorch 1.5.1
cuda 10.1
test on small input tensor (2,8,5,5)

when using the test method in lib/sa/functions to test the speed, I found that the corresponding implementation using pytorch api is much faster than your C code in backward propogation (about 50X faster). Although the forward times of them are relatively close, customized api is slightly faster than torch api.

So why you choose to implement the operation on your own?

linking of pre-model can not open to download

Hi!!! Dr Zhao. It is a very interesting work. I have one Question. Have you tried your model with more layers ?