Comments (9)
I was thinking about using the dimension to decide whether BFGS or L-BFGS should be returned. What would be a suitable borderline value?
from gonum.
Somewhere between 10 and 10000? If the Hessian is basically stationary, and the problem is "small", then BFGS is much preferred. Once BFGS starts to take up a significant amount of the update time (due to the matrix multiplications), then LBFGS is preferred. In theory, if the hessian varies significantly, LBFGS is better because it can adapt to the changing hessian better. In practice, neural networks have a highly non-stationary Hessian, and I have not noticed much difference between the two. A reasonable plan may be to see where the computational time crossover is between BFGS and LBFGS. We still have some optimizations to make in BFGS (using srk2 and dsymv) which should have significant speedups.
from gonum.
Between 10 and 10000, nice :-) OK, let's wait until BFGS is optimized and then run some benchmarks.
from gonum.
Given that the range is the scope of "int", I think I narrowed it down quite a bit.
from gonum.
Has BFGS' use of symmetric matrices been already optimized? Running a small comparison between BFGS and LBFGS gives this:
Dim: 10, BFGS: 94.627µs (20), LBFGS: 61.527µs (20)
Dim: 20, BFGS: 258.911µs (26), LBFGS: 60.82µs (26)
Dim: 30, BFGS: 677.251µs (29), LBFGS: 91.5µs (29)
Dim: 40, BFGS: 1.158941ms (31), LBFGS: 112.179µs (31)
Dim: 50, BFGS: 1.827951ms (33), LBFGS: 134.843µs (33)
Dim: 60, BFGS: 2.626826ms (34), LBFGS: 148.206µs (34)
Dim: 70, BFGS: 3.639132ms (35), LBFGS: 240.221µs (35)
Dim: 80, BFGS: 4.778698ms (36), LBFGS: 202.861µs (36)
Dim: 90, BFGS: 6.130425ms (37), LBFGS: 216.07µs (37)
Dim: 100, BFGS: 7.799486ms (38), LBFGS: 238.968µs (38)
Dim: 200, BFGS: 34.148876ms (43), LBFGS: 448.894µs (43)
Dim: 300, BFGS: 82.53817ms (46), LBFGS: 694.344µs (46)
Dim: 400, BFGS: 156.652703ms (48), LBFGS: 1.359277ms (48)
Dim: 500, BFGS: 273.878261ms (51), LBFGS: 1.233659ms (51)
Objective function is functions.VariablyDimensioned
which is nice because both BFGS and LBFGS need exactly the same number of function evaluations (the number in parentheses).
from gonum.
Is that with cblas or native? The BFGS code has been optimized as far as it is concerned, though some of the routines are slower than they could be in native. I'd guess the right number is somewhere in the 10-30 range, as beyond that the memory requirements go up significantly (and it doesn't seem like speed is much different at the small scales)
from gonum.
That was with native. With OpenBLAS the number are like this:
Dim: 10, BFGS: 151.359µs (20), LBFGS: 59.928µs (20)
Dim: 20, BFGS: 279.296µs (26), LBFGS: 57.422µs (26)
Dim: 30, BFGS: 665.442µs (29), LBFGS: 85.797µs (29)
Dim: 40, BFGS: 1.109582ms (31), LBFGS: 117.449µs (31)
Dim: 50, BFGS: 1.720677ms (33), LBFGS: 129.995µs (33)
Dim: 60, BFGS: 2.433621ms (34), LBFGS: 149.964µs (34)
Dim: 70, BFGS: 3.341203ms (35), LBFGS: 187.494µs (35)
Dim: 80, BFGS: 4.994706ms (36), LBFGS: 205.709µs (36)
Dim: 90, BFGS: 5.673993ms (37), LBFGS: 213.211µs (37)
Dim: 100, BFGS: 7.146321ms (38), LBFGS: 244.066µs (38)
Dim: 200, BFGS: 31.30622ms (43), LBFGS: 487.792µs (43)
Dim: 300, BFGS: 76.301817ms (46), LBFGS: 686.751µs (46)
Dim: 400, BFGS: 145.283187ms (48), LBFGS: 953.991µs (48)
Dim: 500, BFGS: 235.555473ms (51), LBFGS: 1.223314ms (51)
I hope that I did it right because the numbers are almost the same (ok, slightly lower with OpenBLAS). I think that it would be useful if blas/README.md included a simple example on how to use it and how to switch between various implementations. What I did:
package main
import (
"github.com/gonum/blas/blas64"
"github.com/gonum/blas/cgo"
)
func init() {
blas64.Use(cgo.Implementation{})
}
func main() {...}
That's correct, isn't it?
from gonum.
So what about 20?
from gonum.
Yes, that is correct.
from gonum.
Related Issues (20)
- mat: add PivotedQR type
- mat: add type for permutation matrices HOT 1
- all: replace min and max helpers with min/max builtins when go1.21 is lowest supported version
- all: consider using math/rand/v2 when available HOT 1
- LP simplex bug HOT 8
- gonum method like numpy
- graph/path: YenKShortestPaths returns duplicate paths
- fatal error in internal/asm/f32.AxpyInc HOT 1
- proposal: generate unrolled implementations in `internal/asm` HOT 3
- Feature Request: Add DTRSYL3 to gonum or lapack64
- gonum/mat: Dense Matrix Multiplication Bug HOT 1
- a little error in the documentation, where can I fix and PR? HOT 1
- gonum/mat: NewTridiag has arguments flipped HOT 2
- add support for `Lines()` method on Multigraph interface HOT 1
- spatial/r3: `Example_slerp` failure on arm64 HOT 1
- mat/prodcut.go invalid link in the comment HOT 2
- mat: calling qr.Factorize leads to OOM for matrixes with many rows HOT 5
- Dijkstra weight function - return false HOT 1
- all: fix ST1000, ST102[012] errors
- feature request: transitive reduction of a graph HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gonum.