Comments (21)
I should have some free time for this in the next while.
from chainrules.jl.
I think this should be a new package.
Maybe TestingDiffs.jl
or DiffTest.jl
I think it would be good to have this as a test dependency, here and also in other places,
like when ever someone implements ChainRules they should be able to depend on this and test it.
According to @willtebbutt there are many copies of similar code to this, in his repos and also in Nabla.
I think we should make it easiest to test ChainRules, so it should have a ChainRulesCore dependency.
But I think it also might be nice if some of it was a bit more generic, so that it can be used to test orther things, like ZygoteRules (which will be around for a bit, since we don't plan on solving mutation any time soon).
But that is lower priority can can come in a follow on PR after we create it
from chainrules.jl.
I ha e created an empty JuliaDiff/DiffTest.jl repo
from chainrules.jl.
AutoDiffTests might be a better name for the existing package. But even then the names are quite similar. AutoDiffChallenges and DiffTest?
Alternative we could name the new one something like ChainRulesTest / ChainRulesTestUtils. Might be a better way to go.
Naming things is difficult.
(I'm starting to think one package would be fine 😆)
from chainrules.jl.
I guess there are two options here:
- Add a
ChainRules.TestUtil
module, and makeFiniteDifferences
a dependency. - Create another repo called
ChainRulesTestUtil
or something. This might be preferable from a package maintainer's point of view as they could then just addChainRulesCore
to their[deps]
, andChainRulesTestUtil
to their[extras]
, and not depend onChainRules
at all.
because it means that we can probably do something like Zygote's validate flag
This is something that we could think about doing, but I expect that the merits are much less substantial than in an AD-package's case.
from chainrules.jl.
I have limitted time to do ChainRules stuff right now,
In a few weeks will have plenty of time and sometime after that will have both time and people.
What little time I have right now, is direct at the Composite PR in ChainRulesCore, and the Zygote PR.
from chainrules.jl.
Some quick thoughts on this, the second option seems like the most simple route to go down, and it would be nice and clean to just add this new package as a test dependency wherever you might need it.
The third option of a submodule in FDM seems a bit strange to me, as I don't see how the two are directly connected? If we did a new package, are there other utilities in our differential tools that can be pulled out and used across the board?
If so, it would be nice to have a JuliaDiff/DiffTestingUtils
package which could encapsulate all sorts of generic tools and introduce this package as a test dependency in other packages.
from chainrules.jl.
Separate package sounds good to me.
from chainrules.jl.
What a great thread! Time allowing, very happy to help :)
from chainrules.jl.
Option 3.
We could make TestUtil
a submodule of FDM.
We would want to make it a bit more general so you need to pass in that you are using something to get the rules and stuff. (Possibly could be made general enough that you easily use it for testing both chainrules and an AD system). This feels good if we can get a good abstraction.
Maybe something that takes in such a thing and returns a named tuple of closures over it, as a kind of fake module?
from chainrules.jl.
I would be pro- this third option if it could be done in a way that has a simple API that would play nicely with both ChainRules and AD packages. Seems like a common-enough problem.
edit: I guess the pertinent question is how much additional code would one need to implement in ChainRules
or Zygote
for this, on top of whatever we add in FiniteDifferences
.
from chainrules.jl.
Just bumping this. Is anyone keen to work on it? I would love to see it done, as I regularly implement custom adjoints and having something standalone to help out with this would be fantastic. Alas, I don't really have time to work on this at the minute.
from chainrules.jl.
@mattBrzezinski is this a thing you might have time/interest in doing?
from chainrules.jl.
Good point. They probably shouldn't be in the same repo as the intentions of the packages are somewhat different. Jarrett wrote DiffTests many moons ago to constitute a series of quite challenging problems for an AD system to get right (e.g. does a forwards-mode package successfully deal with problems where perturbation confusion could be a problem), whereas we're writing a tool for checking that an AD tool / frule / rrule returns the correct thing.
I think your idea of having different names is probably a good move. We should also reference each package in their respective readmes, explaining what the relationship between the two is to avoid confusion.
from chainrules.jl.
I think ChainRulesTests.jl is a good name, it's explicit and unique enough. I don't have the perms to make it repo, if someone else could that'd be great.
Have modified your permissions :)
from chainrules.jl.
I'm not super against adding a FiniteDifferences.jl dependency to ChainRules.jl
because it means that we can probably do something like Zygote's validate
flag,
where it checked custom sensitivies everytime they are called.
Though perhaps that is better done in the AD only
from chainrules.jl.
I wonder what @willtebbutt thinks?
from chainrules.jl.
Would a new repo be most appropriate? Or fill into https://github.com/JuliaDiff/DiffTests.jl ?
If not, these package names are very very similar probably should pick something different.
from chainrules.jl.
How about a more different name.
The operation we are doing is basically Grad Check
but not just on gradients.
Maybe something super verbose: CheckMyDerviativesAreCorrect.jl
ChainRulesTest is fine though.
from chainrules.jl.
I think ChainRulesTests.jl
is a good name, it's explicit and unique enough. I don't have the perms to make it repo, if someone else could that'd be great.
from chainrules.jl.
https://github.com/JuliaDiff/ChainRulesTestUtils.jl
from chainrules.jl.
Related Issues (20)
- Rules for `kron` HOT 1
- `*(::AbstractVector, ::AbstractMatrix)` pullback triggers scalar indexing on the GPU HOT 1
- ERROR: LoadError: Some tests did not pass: 7126 passed, 7 failed, 4 errored, 0 broken. in expression starting at /home/ian/.julia/packages/ChainRules/Ipuva/test/runtests.jl:19 ERROR: Package ChainRules errored during testing
- Error differentiating composed cross product with Zygote HOT 2
- `@fastmath maximum` broken on 1.10
- Add rules for `LinRange`
- Rule for `typed_vcat`? HOT 1
- Array `getindex` rule unable to handle Zero types and `NotImplemented` HOT 7
- Returning `Broadcasted` cotangents for `Broadcasted` arguments? HOT 7
- `unbroadcast` fails for `Matrix{Tangent}` due to `zero(::Tangent)`
- Missing rules for Tridiagonal HOT 1
- Wrong results for forward-mode `exp!` half of the time HOT 5
- Errors for gradient and hessian of logabsdet of sparse matrix HOT 3
- Missing rrules for `spdiagm`
- Rules for `eachslice` with multiple `dims`
- Reconsider giving `zero` as the derivative of one HOT 2
- arraymath muladd frule has too loose types
- Missing frules for `copy` HOT 1
- Sparse vector to real power throws a pullback error
- `rrule` for `mean(f, x)` is not vectorized? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chainrules.jl.