Comments (39)
I just use the 50 layers resnet from slim.nets.resnet_v2.py:
resnet_v2_50(inputs,
num_classes=None,
is_training=True,
global_pool=False,
output_stride=None,
reuse=None,
scope='resnet_v2_50'):
I achieved 0.99216666666 with batch size 128.
I continue the training with the cutout method(batch size 128), and I achieved 9941666666666666.
Finally, I achieved 0.9955 with batch size 256 at 787150 iter.
I am now looking forward to achieving higher acc with using batch size 1024.
from insightface_tf.
The momentum optimizer increments global step when applying gradients so inc_op is redundant.
from insightface_tf.
I don't know why, so I added Max Batch Size Test
.
from insightface_tf.
the number of weights for logit is 85164*512. It's about 43.6M! But we do not need the logit after training right? I am wondering if it is possible that we save the net only without logit?
from insightface_tf.
Hi, I use the network "tensorflow/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py" instead, and it works! The model size is down to 630M for MS1M and 330M for VGGface2. And I can train the network with batch-size up to 384 with only 1 GTX 1080ti 11G.
from insightface_tf.
Could you test the model size after adding the prelu
and adding the same numbers of Batch Norm
layers as the paper.
from insightface_tf.
I did not read the paper yet, but I will do it after reading it.
I am now interested in implement NASNet and training with your code.
from insightface_tf.
Hi, I added the same numbers of PRelu and Batch_norm layers to resnet_v2.py.
The model size is 640MB with MS1M dataset.
Differences:
1.It seems that there is no gamma in batch_norm in 'tensorflow/tensorflow/contrib/layers/python/layers/layers.py.'
2.There are 14 units in block 3 in L_Resnet_E_IR.py, but 6 units only in block 3 in resnet_v2.py.
3.There is on more conv in each bottleneck in resnet_v2.py.
I notice that there are 2 BN operations between units in L_Resnet_E_IR.py. Maybe we should leave one only.
Below are the details of trainable variables. Note that preact is batch_norm followed by Relu.
Trainable weights in L_Resnet_E_IR.py:
resnet_v1_50/conv1/W_conv2d:0
resnet_v1_50/bn0/beta:0
resnet_v1_50/bn0/gamma:0
resnet_v1_50/prelu0/alphas:0
resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut_conv/W_conv2d:0
resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut_bn/BatchNorm/beta:0
resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut_bn/BatchNorm/gamma:0
resnet_v1_50/block1/unit_1/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block1/unit_1/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block1/unit_1/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block1/unit_1/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block1/unit_1/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block1/unit_1/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block1/unit_1/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block1/unit_2/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block1/unit_2/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block1/unit_2/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block1/unit_2/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block1/unit_2/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block1/unit_2/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block1/unit_2/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block1/unit_3/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block1/unit_3/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block1/unit_3/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block1/unit_3/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block1/unit_3/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block1/unit_3/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block1/unit_3/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut_conv/W_conv2d:0
resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut_bn/BatchNorm/beta:0
resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut_bn/BatchNorm/gamma:0
resnet_v1_50/block2/unit_1/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block2/unit_1/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block2/unit_1/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block2/unit_1/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block2/unit_1/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block2/unit_1/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block2/unit_1/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block2/unit_2/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block2/unit_2/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block2/unit_2/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block2/unit_2/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block2/unit_2/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block2/unit_2/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block2/unit_2/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block2/unit_3/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block2/unit_3/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block2/unit_3/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block2/unit_3/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block2/unit_3/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block2/unit_3/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block2/unit_3/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block2/unit_4/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block2/unit_4/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block2/unit_4/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block2/unit_4/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block2/unit_4/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block2/unit_4/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block2/unit_4/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut_conv/W_conv2d:0
resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut_bn/BatchNorm/beta:0
resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut_bn/BatchNorm/gamma:0
resnet_v1_50/block3/unit_1/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block3/unit_1/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block3/unit_1/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block3/unit_1/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block3/unit_1/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block3/unit_1/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block3/unit_1/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block3/unit_2/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block3/unit_2/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block3/unit_2/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block3/unit_2/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block3/unit_2/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block3/unit_2/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block3/unit_2/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block3/unit_3/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block3/unit_3/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block3/unit_3/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block3/unit_3/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block3/unit_3/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block3/unit_3/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block3/unit_3/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block3/unit_4/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block3/unit_4/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block3/unit_4/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block3/unit_4/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block3/unit_4/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block3/unit_4/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block3/unit_4/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block3/unit_5/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block3/unit_5/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block3/unit_5/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block3/unit_5/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block3/unit_5/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block3/unit_5/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block3/unit_5/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block3/unit_6/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block3/unit_6/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block3/unit_6/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block3/unit_6/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block3/unit_6/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block3/unit_6/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block3/unit_6/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block3/unit_7/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block3/unit_7/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block3/unit_7/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block3/unit_7/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block3/unit_7/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block3/unit_7/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block3/unit_7/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block3/unit_7/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block3/unit_7/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block3/unit_8/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block3/unit_8/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block3/unit_8/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block3/unit_8/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block3/unit_8/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block3/unit_8/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block3/unit_8/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block3/unit_8/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block3/unit_8/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block3/unit_9/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block3/unit_9/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block3/unit_9/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block3/unit_9/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block3/unit_9/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block3/unit_9/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block3/unit_9/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block3/unit_9/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block3/unit_9/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block3/unit_10/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block3/unit_10/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block3/unit_10/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block3/unit_10/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block3/unit_10/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block3/unit_10/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block3/unit_10/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block3/unit_10/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block3/unit_10/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block3/unit_11/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block3/unit_11/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block3/unit_11/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block3/unit_11/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block3/unit_11/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block3/unit_11/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block3/unit_11/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block3/unit_11/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block3/unit_11/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block3/unit_12/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block3/unit_12/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block3/unit_12/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block3/unit_12/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block3/unit_12/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block3/unit_12/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block3/unit_12/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block3/unit_12/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block3/unit_12/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block3/unit_13/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block3/unit_13/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block3/unit_13/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block3/unit_13/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block3/unit_13/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block3/unit_13/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block3/unit_13/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block3/unit_13/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block3/unit_13/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block3/unit_14/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block3/unit_14/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block3/unit_14/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block3/unit_14/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block3/unit_14/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block3/unit_14/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block3/unit_14/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block3/unit_14/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block3/unit_14/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut_conv/W_conv2d:0
resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut_bn/BatchNorm/beta:0
resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut_bn/BatchNorm/gamma:0
resnet_v1_50/block4/unit_1/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block4/unit_1/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block4/unit_1/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block4/unit_1/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block4/unit_1/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block4/unit_1/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block4/unit_1/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block4/unit_2/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block4/unit_2/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block4/unit_2/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block4/unit_2/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block4/unit_2/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block4/unit_2/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block4/unit_2/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/block4/unit_3/bottleneck_v1/conv1_bn1/beta:0
resnet_v1_50/block4/unit_3/bottleneck_v1/conv1_bn1/gamma:0
resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/W_conv2d:0
resnet_v1_50/block4/unit_3/bottleneck_v1/conv1_bn2/beta:0
resnet_v1_50/block4/unit_3/bottleneck_v1/conv1_bn2/gamma:0
resnet_v1_50/block4/unit_3/bottleneck_v1/prelu_layer/alphas:0
resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/W_conv2d:0
resnet_v1_50/block4/unit_3/bottleneck_v1/conv2_bn/BatchNorm/beta:0
resnet_v1_50/block4/unit_3/bottleneck_v1/conv2_bn/BatchNorm/gamma:0
resnet_v1_50/E_BN1/beta:0
resnet_v1_50/E_BN1/gamma:0
resnet_v1_50/E_DenseLayer/W:0
resnet_v1_50/E_DenseLayer/b:0
resnet_v1_50/E_BN2/beta:0
237 trainable weights
Trainable weights in resnet_v2.py(added prelu and BN2)
resnet_v2_50/conv1/weights:0
resnet_v2_50/conv1/biases:0
resnet_v2_50/block1/unit_1/bottleneck_v2/preact/beta:0
resnet_v2_50/block1/unit_1/bottleneck_v2/shortcut/weights:0
resnet_v2_50/block1/unit_1/bottleneck_v2/shortcut/biases:0
resnet_v2_50/block1/unit_1/bottleneck_v2/conv1/weights:0
resnet_v2_50/block1/unit_1/bottleneck_v2/conv1/biases:0
resnet_v2_50/block1/unit_1/bottleneck_v2/BN2/beta:0
resnet_v2_50/block1/unit_1/bottleneck_v2/prelu_layer/alphas:0
resnet_v2_50/block1/unit_1/bottleneck_v2/conv2/weights:0
resnet_v2_50/block1/unit_1/bottleneck_v2/conv2/biases:0
resnet_v2_50/block1/unit_1/bottleneck_v2/conv3/weights:0
resnet_v2_50/block1/unit_1/bottleneck_v2/conv3/biases:0
resnet_v2_50/block1/unit_2/bottleneck_v2/preact/beta:0
resnet_v2_50/block1/unit_2/bottleneck_v2/conv1/weights:0
resnet_v2_50/block1/unit_2/bottleneck_v2/conv1/biases:0
resnet_v2_50/block1/unit_2/bottleneck_v2/BN2/beta:0
resnet_v2_50/block1/unit_2/bottleneck_v2/prelu_layer/alphas:0
resnet_v2_50/block1/unit_2/bottleneck_v2/conv2/weights:0
resnet_v2_50/block1/unit_2/bottleneck_v2/conv2/biases:0
resnet_v2_50/block1/unit_2/bottleneck_v2/conv3/weights:0
resnet_v2_50/block1/unit_2/bottleneck_v2/conv3/biases:0
resnet_v2_50/block1/unit_3/bottleneck_v2/preact/beta:0
resnet_v2_50/block1/unit_3/bottleneck_v2/conv1/weights:0
resnet_v2_50/block1/unit_3/bottleneck_v2/conv1/biases:0
resnet_v2_50/block1/unit_3/bottleneck_v2/BN2/beta:0
resnet_v2_50/block1/unit_3/bottleneck_v2/prelu_layer/alphas:0
resnet_v2_50/block1/unit_3/bottleneck_v2/conv2/weights:0
resnet_v2_50/block1/unit_3/bottleneck_v2/conv2/biases:0
resnet_v2_50/block1/unit_3/bottleneck_v2/conv3/weights:0
resnet_v2_50/block1/unit_3/bottleneck_v2/conv3/biases:0
resnet_v2_50/block2/unit_1/bottleneck_v2/preact/beta:0
resnet_v2_50/block2/unit_1/bottleneck_v2/shortcut/weights:0
resnet_v2_50/block2/unit_1/bottleneck_v2/shortcut/biases:0
resnet_v2_50/block2/unit_1/bottleneck_v2/conv1/weights:0
resnet_v2_50/block2/unit_1/bottleneck_v2/conv1/biases:0
resnet_v2_50/block2/unit_1/bottleneck_v2/BN2/beta:0
resnet_v2_50/block2/unit_1/bottleneck_v2/prelu_layer/alphas:0
resnet_v2_50/block2/unit_1/bottleneck_v2/conv2/weights:0
resnet_v2_50/block2/unit_1/bottleneck_v2/conv2/biases:0
resnet_v2_50/block2/unit_1/bottleneck_v2/conv3/weights:0
resnet_v2_50/block2/unit_1/bottleneck_v2/conv3/biases:0
resnet_v2_50/block2/unit_2/bottleneck_v2/preact/beta:0
resnet_v2_50/block2/unit_2/bottleneck_v2/conv1/weights:0
resnet_v2_50/block2/unit_2/bottleneck_v2/conv1/biases:0
resnet_v2_50/block2/unit_2/bottleneck_v2/BN2/beta:0
resnet_v2_50/block2/unit_2/bottleneck_v2/prelu_layer/alphas:0
resnet_v2_50/block2/unit_2/bottleneck_v2/conv2/weights:0
resnet_v2_50/block2/unit_2/bottleneck_v2/conv2/biases:0
resnet_v2_50/block2/unit_2/bottleneck_v2/conv3/weights:0
resnet_v2_50/block2/unit_2/bottleneck_v2/conv3/biases:0
resnet_v2_50/block2/unit_3/bottleneck_v2/preact/beta:0
resnet_v2_50/block2/unit_3/bottleneck_v2/conv1/weights:0
resnet_v2_50/block2/unit_3/bottleneck_v2/conv1/biases:0
resnet_v2_50/block2/unit_3/bottleneck_v2/BN2/beta:0
resnet_v2_50/block2/unit_3/bottleneck_v2/prelu_layer/alphas:0
resnet_v2_50/block2/unit_3/bottleneck_v2/conv2/weights:0
resnet_v2_50/block2/unit_3/bottleneck_v2/conv2/biases:0
resnet_v2_50/block2/unit_3/bottleneck_v2/conv3/weights:0
resnet_v2_50/block2/unit_3/bottleneck_v2/conv3/biases:0
resnet_v2_50/block2/unit_4/bottleneck_v2/preact/beta:0
resnet_v2_50/block2/unit_4/bottleneck_v2/conv1/weights:0
resnet_v2_50/block2/unit_4/bottleneck_v2/conv1/biases:0
resnet_v2_50/block2/unit_4/bottleneck_v2/BN2/beta:0
resnet_v2_50/block2/unit_4/bottleneck_v2/prelu_layer/alphas:0
resnet_v2_50/block2/unit_4/bottleneck_v2/conv2/weights:0
resnet_v2_50/block2/unit_4/bottleneck_v2/conv2/biases:0
resnet_v2_50/block2/unit_4/bottleneck_v2/conv3/weights:0
resnet_v2_50/block2/unit_4/bottleneck_v2/conv3/biases:0
resnet_v2_50/block3/unit_1/bottleneck_v2/preact/beta:0
resnet_v2_50/block3/unit_1/bottleneck_v2/shortcut/weights:0
resnet_v2_50/block3/unit_1/bottleneck_v2/shortcut/biases:0
resnet_v2_50/block3/unit_1/bottleneck_v2/conv1/weights:0
resnet_v2_50/block3/unit_1/bottleneck_v2/conv1/biases:0
resnet_v2_50/block3/unit_1/bottleneck_v2/BN2/beta:0
resnet_v2_50/block3/unit_1/bottleneck_v2/prelu_layer/alphas:0
resnet_v2_50/block3/unit_1/bottleneck_v2/conv2/weights:0
resnet_v2_50/block3/unit_1/bottleneck_v2/conv2/biases:0
resnet_v2_50/block3/unit_1/bottleneck_v2/conv3/weights:0
resnet_v2_50/block3/unit_1/bottleneck_v2/conv3/biases:0
resnet_v2_50/block3/unit_2/bottleneck_v2/preact/beta:0
resnet_v2_50/block3/unit_2/bottleneck_v2/conv1/weights:0
resnet_v2_50/block3/unit_2/bottleneck_v2/conv1/biases:0
resnet_v2_50/block3/unit_2/bottleneck_v2/BN2/beta:0
resnet_v2_50/block3/unit_2/bottleneck_v2/prelu_layer/alphas:0
resnet_v2_50/block3/unit_2/bottleneck_v2/conv2/weights:0
resnet_v2_50/block3/unit_2/bottleneck_v2/conv2/biases:0
resnet_v2_50/block3/unit_2/bottleneck_v2/conv3/weights:0
resnet_v2_50/block3/unit_2/bottleneck_v2/conv3/biases:0
resnet_v2_50/block3/unit_3/bottleneck_v2/preact/beta:0
resnet_v2_50/block3/unit_3/bottleneck_v2/conv1/weights:0
resnet_v2_50/block3/unit_3/bottleneck_v2/conv1/biases:0
resnet_v2_50/block3/unit_3/bottleneck_v2/BN2/beta:0
resnet_v2_50/block3/unit_3/bottleneck_v2/prelu_layer/alphas:0
resnet_v2_50/block3/unit_3/bottleneck_v2/conv2/weights:0
resnet_v2_50/block3/unit_3/bottleneck_v2/conv2/biases:0
resnet_v2_50/block3/unit_3/bottleneck_v2/conv3/weights:0
resnet_v2_50/block3/unit_3/bottleneck_v2/conv3/biases:0
resnet_v2_50/block3/unit_4/bottleneck_v2/preact/beta:0
resnet_v2_50/block3/unit_4/bottleneck_v2/conv1/weights:0
resnet_v2_50/block3/unit_4/bottleneck_v2/conv1/biases:0
resnet_v2_50/block3/unit_4/bottleneck_v2/BN2/beta:0
resnet_v2_50/block3/unit_4/bottleneck_v2/prelu_layer/alphas:0
resnet_v2_50/block3/unit_4/bottleneck_v2/conv2/weights:0
resnet_v2_50/block3/unit_4/bottleneck_v2/conv2/biases:0
resnet_v2_50/block3/unit_4/bottleneck_v2/conv3/weights:0
resnet_v2_50/block3/unit_4/bottleneck_v2/conv3/biases:0
resnet_v2_50/block3/unit_5/bottleneck_v2/preact/beta:0
resnet_v2_50/block3/unit_5/bottleneck_v2/conv1/weights:0
resnet_v2_50/block3/unit_5/bottleneck_v2/conv1/biases:0
resnet_v2_50/block3/unit_5/bottleneck_v2/BN2/beta:0
resnet_v2_50/block3/unit_5/bottleneck_v2/prelu_layer/alphas:0
resnet_v2_50/block3/unit_5/bottleneck_v2/conv2/weights:0
resnet_v2_50/block3/unit_5/bottleneck_v2/conv2/biases:0
resnet_v2_50/block3/unit_5/bottleneck_v2/conv3/weights:0
resnet_v2_50/block3/unit_5/bottleneck_v2/conv3/biases:0
resnet_v2_50/block3/unit_6/bottleneck_v2/preact/beta:0
resnet_v2_50/block3/unit_6/bottleneck_v2/conv1/weights:0
resnet_v2_50/block3/unit_6/bottleneck_v2/conv1/biases:0
resnet_v2_50/block3/unit_6/bottleneck_v2/BN2/beta:0
resnet_v2_50/block3/unit_6/bottleneck_v2/prelu_layer/alphas:0
resnet_v2_50/block3/unit_6/bottleneck_v2/conv2/weights:0
resnet_v2_50/block3/unit_6/bottleneck_v2/conv2/biases:0
resnet_v2_50/block3/unit_6/bottleneck_v2/conv3/weights:0
resnet_v2_50/block3/unit_6/bottleneck_v2/conv3/biases:0
resnet_v2_50/block4/unit_1/bottleneck_v2/preact/beta:0
resnet_v2_50/block4/unit_1/bottleneck_v2/shortcut/weights:0
resnet_v2_50/block4/unit_1/bottleneck_v2/shortcut/biases:0
resnet_v2_50/block4/unit_1/bottleneck_v2/conv1/weights:0
resnet_v2_50/block4/unit_1/bottleneck_v2/conv1/biases:0
resnet_v2_50/block4/unit_1/bottleneck_v2/BN2/beta:0
resnet_v2_50/block4/unit_1/bottleneck_v2/prelu_layer/alphas:0
resnet_v2_50/block4/unit_1/bottleneck_v2/conv2/weights:0
resnet_v2_50/block4/unit_1/bottleneck_v2/conv2/biases:0
resnet_v2_50/block4/unit_1/bottleneck_v2/conv3/weights:0
resnet_v2_50/block4/unit_1/bottleneck_v2/conv3/biases:0
resnet_v2_50/block4/unit_2/bottleneck_v2/preact/beta:0
resnet_v2_50/block4/unit_2/bottleneck_v2/conv1/weights:0
resnet_v2_50/block4/unit_2/bottleneck_v2/conv1/biases:0
resnet_v2_50/block4/unit_2/bottleneck_v2/BN2/beta:0
resnet_v2_50/block4/unit_2/bottleneck_v2/prelu_layer/alphas:0
resnet_v2_50/block4/unit_2/bottleneck_v2/conv2/weights:0
resnet_v2_50/block4/unit_2/bottleneck_v2/conv2/biases:0
resnet_v2_50/block4/unit_2/bottleneck_v2/conv3/weights:0
resnet_v2_50/block4/unit_2/bottleneck_v2/conv3/biases:0
resnet_v2_50/block4/unit_3/bottleneck_v2/preact/beta:0
resnet_v2_50/block4/unit_3/bottleneck_v2/conv1/weights:0
resnet_v2_50/block4/unit_3/bottleneck_v2/conv1/biases:0
resnet_v2_50/block4/unit_3/bottleneck_v2/BN2/beta:0
resnet_v2_50/block4/unit_3/bottleneck_v2/prelu_layer/alphas:0
resnet_v2_50/block4/unit_3/bottleneck_v2/conv2/weights:0
resnet_v2_50/block4/unit_3/bottleneck_v2/conv2/biases:0
resnet_v2_50/block4/unit_3/bottleneck_v2/conv3/weights:0
resnet_v2_50/block4/unit_3/bottleneck_v2/conv3/biases:0
resnet_v2_50/postnorm/beta:0
resnet_v2_50/DenseLayer/W:0
resnet_v2_50/DenseLayer/b:0
resnet_v2_50/BatchNorm/beta:0
158 trainable weights
from insightface_tf.
Hi, I just found that the first conv and bn should be followed by a 3x3 stride2 pooling layer, but it is not in L_Resnet_E_IR.py.
After adding the pooling layer, the model size for MS1M is down to 1.31G and 1.08G for VGG2.
I finally can train L_Resnet_E_IR.py with batch_size 128 on VGG2(MS1M not sure), not only batch_size 16.
However, the training speed is one-fifth of resnet_V2.py.
By the way, the number of trainable weights is about 141M in L_Resnet_E_IR.py, and about 40M in resnet_V2.py.
from insightface_tf.
@HsuTzuJen
Thanks for you compare the different models.
from insightface_tf.
The implemented architecture of L_ResnetE_IR is as mentioned in the paper? As you @HsuTzuJen said there are some differences. Specially the size of implemented model is big.
from insightface_tf.
The paper says stride of the second conv2 in each bottleneck is 2, but those in implementation are 1 except for those in the first unit of each block. Is it OK?
from insightface_tf.
@TengliEd I am using ResNetV2, and I achieve acc 0.995 test on lfw now. I think L_ResnetE_IR is improved from ResNet, but I have not tried to train it more yet since the model is too big. I will try to implement L_ResnetE_IR myself and train it.
I think the model size of L_ResnetE_IR should be similar to ResNetV2 and it is about 160MB.
from insightface_tf.
So cool @HsuTzuJen. You got exact architecture of L_ResNet100E_IR? I am still confused about the structure of Resnet part. Plus, please don't forget to mention me as soon as you finish your own implementation. Cheers!
from insightface_tf.
@TengliEd I think you can read the paper about ResNet first"https://arxiv.org/abs/1512.03385".
There are net models in "https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/slim/python/slim/nets",
and you can try to revise the code. It's more simple to implement in this way.
from insightface_tf.
Thanks @HsuTzuJe. I have read the paper but the Arcface implementation here seems not to follow the original structure. For instance, "There are 14 units in block 3 in L_Resnet_E_IR.py, but 6 units only in block 3 in resnet_v2.py." you said. Arcface not only modified each unit bottleneck but also the number of units, is it true?
from insightface_tf.
@TengliEd Since each unit bottleneck is modified, the number of units should be too.
They count layers by the number of convolution layer.
There are 2 conv in each unit for L_Resnet_E_IR but 3 conv in each unit for ResNetV2.
from insightface_tf.
I just compared the two models:
ResNetV2(50 layers): 16 3x3 convs and 32 1x1 convs in 4 blocks.
L_Resnet_E_IR(50 layers): 48 3x3 convs in 4 blocks.
The size of one 3x3 conv is 9 times to one 1x1 conv, and there are more batchnorm is the L_Resnet_E_IR than ResNetV2. Due to these reasons, it makes sense that the size of L_Resnet_E_IR is much bigger
than ResNetV2.
By the way, the size of the pre-trained L_Resnet_E_IR (50 layers) provided by insightface (mxnet version) is only 160MB which is closed to ResNetV2(50 layers, tf version). I am curious what makes them so different since they are in the same architecture(L_Resnet_E_IR implement in tf and mxnet).
from insightface_tf.
You are sure L_ResNet_E_IR was implemented in TF the same way as in MxNet?
from insightface_tf.
@HsuTzuJen How is your own implementation going? Is there any changes to this one? :)
from insightface_tf.
@TengliEd NO, I got the same result just like L_Resnet_E_IR.py.
I think that we should try the insightFace(mxnet), but I do not know about mxnet.
from insightface_tf.
@HsuTzuJen I just checked the MxNet code. There is no bottleneck in L_ResNet_E_IR. So the filter number in each unit of a block is same. No need to 4* base_depth.
from insightface_tf.
@TengliEd Thank you very much. I just check it and I found that the size of the model is only 166MB without 4* base_depth.
@auroua I think we can use your code to achieve good performance now!
from insightface_tf.
My L_ResNet100_E size is 402MB and L_ResNet50_E is 320 MB. How can you achieve just 166MB?
from insightface_tf.
It is 320MB that the model is included the logit layer"arcface_loss(embedding, labels, out_num, w_init=None, s=64., m=0.5)"(it is a fully connected layer for softmax, and it is useful only in training) and L_ResNet50_E , but it is 166MB with L_ResNet50_E only.
I only save the trained L_ResNet50_E since we test the model with only L_ResNet50_E.
from insightface_tf.
OK. Do you use WeChat or other similar apps? I wanna ask you for some details about how to get 99.5%+ accuracy on LFW. I only got 99.02%. :)
from insightface_tf.
Or plz provide your train_nets.py and other modified parts. Thx
from insightface_tf.
@HsuTzuJen Your result is better when using no bottle neck?
from insightface_tf.
How can you achieve so large batch size? I am using p100 12G GPU and the maximum batch size is only 50 when employing L_ResNet_100E_IR. Besides, you mean dropout by cutout method?
from insightface_tf.
I have a GPU server with 4 1080ti.
I do not use the dropout.
Cutout is an image preprocessing method which is easy to implement.
The paper : https://arxiv.org/abs/1708.04552
from insightface_tf.
Thx @HsuTzuJen. I also have a server with 4 GPUs but don't know how to boost batch size. It seems there is no difference in maximum batch size between 4 GPUs and 1 GPU. Can you share your code to let me repeat your result? Cheers!
from insightface_tf.
Only one gpu memory is fully used. I have set coda_visible_devices to 0,1,2,3. You use train_nets.py to train your model? How can I fully use the gpu memory?
from insightface_tf.
Please use train_nets_mgpu.py set parser.add_argument('--num_gpus', default=4, help='the num of gpus')
from insightface_tf.
OK. parser.add_argument('--batch_size', default=32, help='batch size to train network') means actual batch size is 4*32?
from insightface_tf.
The train_nets_mgpu.py
have some bugs, you should do some modify before using this code. About how to modify the code, you could reference the code train_nets.py
.
from insightface_tf.
The added lines used to train on multiple gpus based on train_nets.py are correct? I suppose you mean the accuracy computation line. When I need a 128 batch size, I just need to set batch_size to 32 on here?
from insightface_tf.
Just set 128 if you want a 128 batch size, and it would be split into num_gpu parts.
from insightface_tf.
OK. I got it. You use other tricks like lr schedule?
from insightface_tf.
Just took a look at L_Resnet_E_IR_fix_issue9.py and it appears that both dropout and batch normalization are kept on during testing. In particular batch norm is being passed is_train=True as a hard parameter throughout. Is this intentional? If I've understood correctly it means you're actually going Adaptive Batch Normalization, which is a domain adaptation technique.
from insightface_tf.
Related Issues (20)
- About the accuracy computation(关于计算准确率的问题) HOT 1
- the number of samples is not 5.8M
- resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut_bn/BatchNorm/beta
- stom
- could I get the 512-D face embedding features ?
- pre trained models HOT 2
- How can I directly test the LFW data set with an existing model to get accuracy?Which folder is the model in?
- Embeddings model d
- In what format does the labels should be
- How to download pretrain ?
- Thank you for your code,could you tell why I feed input with shape(n,112,112,3),but just get the embedding with shape(1,512) HOT 1
- Performance issues in your project (by P3) HOT 1
- Performance issues in train_nets.py(P2)
- from models import base_server NOT IMPORTING
- center parallel strategy
- any instruction how to start?
- Share model please,,..
- Did you try a lighter base such as MobileFaceNet?
- compared with facenet HOT 2
- How to solve the problem of "summary" errors during training?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from insightface_tf.