XNOR Net uses input quantization of ste_sign about zoo HOT 5 CLOSED

Hi, great to hear you find larq & larq-zoo useful!

You are correct, the default implementation doesn't use scaling of the inputs. In our experiments, we found that scaling the inputs did not improve accuracy, so we decided to leave it out.

As a side note, in our recent work on BNN optimization (paper, code) we found we could improve accuracies without using any scaling factors (neither activations or weights).

from zoo.

The best way to implement this is probably to follow equation 11/figure 2 in the paper, i.e. something like:

def xnor_conv(x, kernel_size=3):
    A = tf.reduce_mean(tf.abs(x), axis=[3], keepdims=True)
    k = tf.ones([kernel_size, kernel_size, 1, 1])* 1/(kernel_size**2)
    K = tf.nn.conv2d(A, k, 1, "SAME")
    x = lq.layers.QuantConv2D(32, kernel_size, padding="SAME", input_quantizer="ste_sign", kernel_quantizer="ste_sign", kernel_constraint="weight_clip")(x)
    x = x * K
    return x

We would have to revisit this before sharing results as it has been a while ago. At this point I don't think we will do that anytime soon, XNOR-net was a bit of a dead end for us and indeed this type of scaling adds complexity to the model. But would be curious to hear if you have better luck using input scaling.

from zoo.

Thanks for the quick response.

I have a couple of questions regarding the same.

I have read in other research papers stating that the input scaling is expensive, but I couldn't find any reference that says scaling inputs would not improve accuracy. Would it be possible for your team to share the results of this experiment which led to ignoring the input scaling factor?.

Also how would one recreate the exact XNOR network with input scaling using the Larq framework?

from zoo.

Thanks a lot for the clarification.
I have already read BNN optimization (paper, code) and found it to be a really interesting view on BNN's.

from zoo.

You are correct, the default implementation doesn't use scaling of the inputs. In our experiments, we found that scaling the inputs did not improve accuracy, so we decided to leave it out.

Hi!
Is there a comparison of scaling factors I could cite? I have a similar result, I wonder if it is a known fact.

from zoo.