Comments (4)
Yes, gradient tape computes gradients, but in forward mode we do not compute gradients (at least in regular Neural ODEs), we just integrate NN with some selected solver: odeint(net, h0, t_steps)
. However in backward mode we have to compute jacobian of NN w.r.t its inputs (see the snippet from my previous answer) to accumulate gradients using adjoint method, but note we do it once in each backward step and we don't need to create and memorize whole computational graph. To summarize in forward mode, we solve ODE by integrating NNetwork with selected solver. Then one must define some loss function which generates gradients. These gradients are then plug in into adjoint ODE which additionally requires evaluation of the jacobian-vector-product (JVP): a(t)^T * dNet(h)/dh
at every step of the solver.
from deep-learning-notes.
Yes, adjoint method is implemented and it requires a gradient computation of Neural Net output w.r.t to its input. You are probably asking about this part of code:
def _backward_dynamics(self, state):
t = state[0]
ht = state[1]
at = -state[2]
with tf.GradientTape() as g:
g.watch(ht)
ht_new = self._model(inputs=[t, ht])
# compute at^T df/dz, at^T df / dtheta c.f aug_dynamics from paprt
gradients = g.gradient(
target=ht_new, sources=[ht] + self._model.weights,
output_gradients=at
)
return [1.0, ht_new, *gradients]
This is exactly the same what aug_dynamics
function from Algorithm 1 from paper. Is it now clear ?
from deep-learning-notes.
Just a little confused, doesn't the tf.gradient tape calculate the gradients? Is this the gradient during forward mode differentiation? I'll brush up on the paper once again. Just these couple of doubts.
from deep-learning-notes.
Thank you. That cleared it out.
from deep-learning-notes.
Related Issues (13)
- Question about 2017-09-Poincare-Embeddings HOT 1
- License? HOT 1
- NeuralODE with multiple features? HOT 3
- Minor typo in normalizing flows slides HOT 2
- Questions about Glow implementation (Not bugs) HOT 8
- Curious about Glow implementation: some weights look frozen? HOT 9
- Do you have Bibtex? HOT 5
- Using neural ODE to estimate dynamics of forced systems
- Can't find module "neural_ode" in 2.Demo_optimize_bullet_trajectory.ipynb HOT 2
- SGD in Max-Norm HOT 3
- bugs in CIFAR10 training with ResNet-32 HOT 2
- About oversampling-datasets-example.ipynb HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deep-learning-notes.