Comments (4)
anicolson
commented on May 20, 2024
Hi,
The layer normalisation typically employed in networks is sequence-wise layer normalisation. The statistics are thus calculated over all the features of a frame and over the entire sequence. So you are right, layer normalisation is non-causal.
However, 3f is using frame-wise layer normalisation. The statistics are calculated only over the features of the frame for frame-wise layer normalisation. This means that no features from future frames are used for the current frame, giving us a causal form of layer normalisation.
Hope this helps.
…________________________________
From: cohendrake <[email protected]>
Sent: Tuesday, 21 January 2020 1:37 PM
To: anicolson/DeepXi <[email protected]>
Cc: Subscribed <[email protected]>
Subject: [anicolson/DeepXi] Is ResNet 3f causal？ (#18)
Good job. As layer normalization is widely used in the ResNet 3f, I doubt that it's a causal network—‘future’ features are actually included. I've tried to remove all the layer normalizations and the results turn out to be much worse.
Is layer normalization dispensable?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<#18?email_source=notifications&email_token=AGHGZ7SUVNUCBWUTBV3ZJZDQ6ZUXJA5CNFSM4KJM4SK2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IHQDEIA>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AGHGZ7TEQ657AIJ4KEWXXZTQ6ZUXJANCNFSM4KJM4SKQ>.
from deepxi.
cohendrake
commented on May 20, 2024
Thanks! Do you mean this causal form of layer normalization is applied over different dimensions of the feature for the current frame?
from deepxi.
anicolson
commented on May 20, 2024
So the statistics are calculated over all of the dimensions of the current frame. e.g. over the 257 frequency bins of the current frame, giving a mean and variance both of size 1. Each of the frequency bins is then normalised using the same mean and variance.
Hope this helps
from deepxi.
cohendrake
commented on May 20, 2024
Got it.
from deepxi.
Related Issues (20)
- Multi gpu trainin HOT 4
- Understanding the loss implementation HOT 9
- can i train this model without gpu? HOT 1
- How to run version resnet-1.0n? HOT 1
- Errors during training HOT 2
- What is the major consideration when choosing Hamming window with no periodic? HOT 11
- Clarity on file format reqruired in '/set' folder HOT 1
- no deepxi.network.attention.py file thus no MHANet class HOT 2
- Running Inference/Testing on Multiple GPUs HOT 7
- Training on other noise dataset gives resuts worse than unprocessed noisy HOT 5
- Denoise Live Microphone Feed HOT 1
- I can't find the implementation of addnoise function from deep_xi_test_set.m HOT 4
- Some questions about the data and the training process HOT 5
- Error loading pretrain model during inference HOT 3
- i can not find the freesound packs. does it mean Sound Ids HOT 2
- Some questions HOT 2
- Additional questions HOT 7
- mhanet loss results
- Can the MHANet run in real time HOT 4
- ValueError: No sample.npz file exists. (Inference)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepxi.