jutho / optimkit.jl Goto Github PK

I want to cite this package in my article.
As well as TensorOperations.jl, DOI with ZENODO is helpful.

This package's test suite generates a lot of output, often triggering the PkgEval log limit. For example: https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_date/2022-11/19/OptimKit.primary.log

It would be nice if by default the logs weren't as verbose 🙂

Sorry to sound like a broken record and I know we discussed in #10 but there's still an infinite loop path in bisect. I confirm because I run into it sporadically, and editing this line to read if c.dϕ >= 0 || numfg>10 to break out if it tries more than 10 bisect steps fixes it for me.

No doubt I have some numerically noisy gradients causing this, and I'm happy if the package sends me a warning or even errors when bisect fails due to that, but just infinite looping really kills my analysis (eg I run 100 sims and step away from my computer for a while, only to come back and find sim 3 got stuck in an infinite loop and it never went further).

First, thanks a ton for this package, it really fills in exactly the missing piece for me of other Julia optimization packages that only really work with Array. However, at the moment OptimKit seems to only work with Float64, e.g.

julia> using OptimKit

julia> optimize(x->(x'x,2x), ones(Float32,5), LBFGS())
ERROR: MethodError: no method matching OptimKit.HagerZhangLineSearchIterator(::var"#1#2", ::typeof(OptimKit._retract), ::typeof(OptimKit._inner), ::OptimKit.LineSearchPoint{Float32,Array{Float32,1},Array{Float32,1}}, ::Array{Float32,1}, ::Float64, ::Bool, ::HagerZhangLineSearch{Float64})

Would be great if OptimKit could work with any eltype <:Real (not just Float32, but BigFloat, etc...). I unfortunately don't have the time to make a proper PR, but if its helpful, I have it working with Float32 here: marius311@ce2136a where with this you can do

optimize(x->(x'x,2x), ones(Float32,5), LBFGS(gradtol=1f-8))

and it works.

I am not sure if this is intentional or not, but all the optimisation algorithms exit with an error if the initial guess was exactly at the optimum. The simplest example:
using OptimKit; alg=GradientDescent(;); fg(x)=sum(x.^2), 2.0 .*x; optimize(fg, [0.],alg);
This code terminates with an error "linesearch was not given a descent direction!". The same for the conjugate gradient algorithm. The LBFGS algorithm tries to start the linear search with d $d\phi_0=NaN$ and stalls without an error message.

For complex optimization, there is an error for missing method isless(::ComplexF64, ::ComplexF64). Is the complex number supported in OptimKit?

It would be helpful if one could add the option of terminating the optimization procedure in case a linesearch is not able to find a new set of variables that satisfy the Wolfe conditions.
Do you think that this could be included?
This could be done by giving the information about the success of the linesearch into the finalize! function such that the user can use it to terminate the optimization in that case.
Best wishes,
Erik

One other minor thing, with a sufficiently poorly conditioned problem, it seems its possible to get stuck in an infinite loop in bisect here (at least it happened to me).

I can try and get a MWE if needed, let me know, but perhaps a simple fix is to add a maxiter type argument there as well, or simply use the same one from the HagerZhangLineSearch ?

Hello,
Can I use ConjugateGradient with Float32 function value and gradient? I have been running calculations with Float64 and everything has been fine, but with Float32 I get the following error:

MethodError: no method matching OptimKit.LineSearchPoint(::Float32, ::Float32, ::Float64, ::ITensor, ::Float32, ::ITensor, ::ITensor)
Closest candidates are:
  OptimKit.LineSearchPoint(::T, ::T, ::T, ::X, ::T, ::G, ::G) where {T<:Real, X, G} at /mnt/home/astrashko/.julia/packages/OptimKit/xpmbV/src/linesearches.jl:6

This is really just for my curiosity. If appropriate, can you include a comment about why not just _inner(x, v1, v2) = LinearAlgebra.dot(v1, v2)? I can't tell if it's about accuracy, or performance, or ....

I know that LinearAlgebra.generic_norm2 does some rescaling for better numerical accuracy, and maybe dot does not.