training with nengo-dl about nengo-dl HOT 4 CLOSED

I notice that you're using LIF neurons, which are probably significantly different than what you were using in Keras. I'd start out by using the same nonlinearity as you were using before (probably nengo.RectifiedLinear) and try to replicate the original performance, to make sure that the model has been accurately translated to the Nengo syntax. And then gradually start changing elements like the nonlinearity to see how that impacts performance.

from nengo-dl.

Hi Daniel,

Thank you for your reply. Do you know why the training in Nengo takes more time than in DNN frameworks? The model I used in Keras takes about 2 minutes to train one epoch, the same model in Nengo takes up to 4 hours to finish one epoch. Now I changed the neuron type to Rectified Linear to make it match with the "Relu" activation I used in Keras. The softmax loss I observed in Nengo is still very different than the loss from Keras. Does the model still consider as SNN models after changing the neuron type to Relu?

from nengo-dl.

Do you know why the training in Nengo takes more time than in DNN frameworks? The model I used in Keras takes about 2 minutes to train one epoch, the same model in Nengo takes up to 4 hours to finish one epoch

We'd expect Nengo to be a little bit slower than a DNN framework when doing DNN tasks, since Nengo is optimized for simulating SNN networks (in particular, simulating networks over time), as opposed to DNN frameworks, which are optimized for simulating standard DNNs. However, that difference should be on the order of 1-2x, not like the times you're seeing.

When I run your code, it takes about 1.5 minutes per epoch, which seems more in line with what you were expecting.

If I had to guess, I'd say that you're accidentally running your training on the CPU, rather than GPU. Make sure that you have tensorflow-gpu installed in whatever environment you're using, not tensorflow.

The softmax loss I observed in Nengo is still very different than the loss from Keras. Does the model still consider as SNN models after changing the neuron type to Relu?

When using relus the output should be exactly the same as in Keras (assuming your keras model was using relu's as well). If the output is not the same, that tells us that something about the original keras model has not been translated into nengo-dl. Perhaps some input pre-processing?

One thing you could try, to help with debugging your model, is to take your entire Keras model and embed it inside a TensorNode (like in this example https://www.nengo.ai/nengo-dl/examples/tensorflow-models.html). Make sure that you get the same performance there as in Keras, then you'll know that you've captured everything about the input/output processing in that model. Then you can gradually start separating the model out into multiple tensor_layers, and make sure that performance remains unchanged.

from nengo-dl.

Closing this since the general questions are answered I think. Feel free to reopen if that is not the case!

from nengo-dl.