Comments (4)
Sorry for the delay!
-
[11] referred to https://arxiv.org/abs/1902.06720, sorry for the confusion - at some point we replaced references with inline links but missed this one, will fix.
-
Yes, absolutely, you can parameterize your kernels and backprop through
nt.predict
functions, and, in general, most othernt
functions. -
Same for NTK, with one caveat that interpreting NTK inference as a GP is a bit nuanced. Running gradient descent on an infinitely wide neural network yields a multivariate normal distribution as eq 16 in https://arxiv.org/pdf/1902.06720.pdf (the one you get by passing
), which is not the same as a GP posterior using an NTK kernel (the one you get by passingget="ntk"
toget="ntkgp"
). I recommend @bobby-he's paper https://proceedings.neurips.cc/paper/2020/file/0b1ec366924b26fc98fa7b71a9c249cf-Paper.pdf for details on this.
Re MLE for NTK:
- If you want to optimize the marginal likelihood of your training set, then it is the same for both NNGP and NTK - NTK at initialization, before training, is exactly the NNGP.
- If you want to optimize the validation loss or the predictive likelihood on your validation data given training data, then NNGP and NTK will be different - see equations (13 - nngp) and (16 - ntk) in https://arxiv.org/abs/1902.06720. But both are Gaussians (with different means and covariances that you can get through https://neural-tangents.readthedocs.io/en/latest/_autosummary/neural_tangents.predict.gradient_descent_mse_ensemble.html#neural_tangents.predict.gradient_descent_mse_ensemble) and you can backprop through their validation loss / likelihood.
Btw https://arxiv.org/abs/2012.09943 seems to try something related - tune neural network parameters by optimizing the marginal likelihood of the respective NNGP.
Lmk if this helps, I'm not sure if I answered / understood everything correctly!
from neural-tangents.
Hi thanks for detailed answer i will look into the material. And a quick follow up so why don't we optimize
from neural-tangents.
I think https://arxiv.org/abs/2012.09943 looks into those. You're right it's possible to optimize these parameters and sometime ago I experimented with it a bit, but just didn't see much generalization improvement. I think the reason is that these are only 2 scalar parameters per layer, and good defaults for them already been been studied quite a lot (e.g. https://arxiv.org/pdf/1611.01232.pdf style papers), so I just don't think they provide enough flexibility to get a noticeable improvement. But if you parameterize your kernel with lots of parameters, I think it's much more promising. I believe this paper https://arxiv.org/abs/2102.03909 got improvement in meta-learning settings from optimizing the initialization parameters of a neural network by tuning them via the empirical (https://neural-tangents.readthedocs.io/en/latest/empirical.html) NTK, so in this setting parameters of the kernel = parameters of a finite width neural network.
from neural-tangents.
Thanks alot your answer was very helpful !
from neural-tangents.
Related Issues (20)
- The analytical output of GP can not fit the result of NNGP generated by the nt.predict.gp_inference HOT 1
- Question: Relu Kernel Computation HOT 3
- Question regarding OOM issues HOT 3
- Question regarding lr in Neural Tangents Cookbook
- eNTK implementation uses deprecated xla attribute HOT 2
- Colab notebooks issue HOT 2
- How to obtain aleatoric uncertainty? HOT 2
- How to compute the empirical after kernel? HOT 1
- pip install issues HOT 2
- Erf function goes beyond [-1,1] HOT 2
- using stax.Cos(a=1.0, b=1.0, c=0.0) to get kernel from conv layer gives error HOT 2
- NTK is not PD
- stax.serial PSDness HOT 1
- How to use batch to gradient_descent_mse_ensemble ? HOT 1
- NTK/NNGP behavior in the infinite regime when weights are drawn from Gaussians with high standard deviation HOT 7
- NKT_mean output Nan, when the number of training sample is increased HOT 3
- Inefficient jacobian computation for embedding layers. HOT 1
- Question regarding the cookbook
- Calling the empirical kernel function with different parameters returns same result
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from neural-tangents.