Comments (6)
Hello,
If you are talking about means during data normalization - this is another parameters that calculated only once.
If we are considering mean and std at the batch norm - they are updated every training step, based on training flag settings. So everything should be done under the hood.
Of course, it's possible that tensorflow API was changed a little bit - so code should be rechecked.
from vision_networks.
@ikhlestov No, I mean that you have to explicitly update moving mean and moving variance in Batch Normalization before applying gradients on trainable variables. Something is like below:
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
train_op = optimizer.minimize(loss)
from vision_networks.
Thank you for your attention, I'll rewrite code
from vision_networks.
@TengliEd Oh, actually I remember this case - I just set updates_collections=None to force the updates in place, so there is no reason to call update with control_dependencies
from vision_networks.
@ikhlestov Yes, thanks for giving me another workaround to update them. But I still got confused about puting weight decay on all trainable variables. l2_loss = tf.add_n([tf.nn.l2_loss(var) for var in tf.trainable_variables()])
. You know we need to decay only weights not bias.
from vision_networks.
As far as I know tensorflow.nn.conv2d layer doesn't contain bias. And the whole network based on that layers - so actually there is no biases.
In any case the training results show nearly the same performance as in the authors implementation and paper - so there is no reason to tweak anything.
It you want - you can make a fork and implement any updates.
from vision_networks.
Related Issues (20)
- can you provide your trained models? HOT 5
- SVHN normalization issue HOT 8
- How about performance for flower dataset? HOT 1
- The order of image normalization and augmentation seems to be the same as the original implementation? HOT 5
- Problem regarding the number of features generated at the initial convolution layer HOT 1
- requirements issue HOT 1
- How to use 2 gpus? HOT 1
- Problem about the running time. HOT 2
- l2 regularization for bias too.. is it necessary? HOT 1
- Train on custom dataset HOT 1
- Train on images on my desktop HOT 1
- Choose GPU device HOT 9
- ResourceExhaustedError: OOM when allocating tensor HOT 2
- Why did you remove the max pooling layer after the initial convolution? HOT 1
- 'by_chanels' normalization issue between different splits HOT 3
- Pre-trained weights HOT 1
- Is there some way to get the softmax (probability values) from the saved models using the repos code. HOT 1
- batch_norm in dense_net.py HOT 1
- Error while running SVHN Dataset
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vision_networks.