pointer being freed was not allocated about ipopt.jl HOT 33 CLOSED

@ferrolho
I recommend to use the Ipopt_jll.jl precompiled with Yggdrasil and use HSL.jl temporary to compile coinhsl.
Ipopt and HSL use libgfortran and they must be compiled with the same version.
METIS 4 is also not working on the new Apple M1...
Therefore, you cannot use MA57.

If you can wait a little bit, the best solution is to wait the release of JuliaHSL / HSL_jll.jl.
I also added the support of METIS 5 with JuliaHSL.
I hope that we can release them this week such that it solves all issues.

from ipopt.jl.

I have now tried JuliaHSL, and it also worked really well! 🎉 I have taken the opportunity to edit the instructions in the README.md slightly — see #367.

Since I am using a MacBook Air, I had to explicitly delete the quarantine attribute with

xattr -d com.apple.quarantine lbt_HSL_jll.jl-2023.5.26.zip

otherwise, I get the following error:

I was then able to run the following test code snippet without issues:

import Pkg
Pkg.develop(path="/Users/henrique/Downloads/HSL_jll.jl-2023.5.26")

using JuMP, Ipopt
import HSL_jll

model = Model(Ipopt.Optimizer)
set_attribute(model, "linear_solver", "ma57")
@variable(model, x)
@objective(model, Min, x^2)
optimize!(model)

Thank you again both for your help!

from ipopt.jl.

Thanks for for your comment @ferrolho.
I updated the documentation of JuliaHSL to add the prefixes lbt_ and openblas_ in the commands with xattr.
Do we also need to remove the quarantine with the archives JuliaHSL-2023.5.26.tar.gz and JuliaHSL-2023.5.26.zip?

from ipopt.jl.

The home-brew binaries are not officially supported by us, and you're unlikely to get much help debugging. Any number of things could be going wrong.

for exporting the path to the Ipopt that I built with coinbrew, and which allows me to use different linear solvers besides MA27.

This is not true. You can use any linear solver with the default binaries that we provide by following https://github.com/jump-dev/Ipopt.jl#linear-solvers.

from ipopt.jl.

for exporting the path to the Ipopt

Did you follow the instructions here: https://jump.dev/JuMP.jl/stable/developers/custom_solver_binaries/?

Setting DYLD_LIBRARY_PATH isn't sufficient to use your binaries. It's probably the cause of the segfault.

from ipopt.jl.

The home-brew binaries are not officially supported by us, and you're unlikely to get much help debugging. Any number of things could be going wrong.

I understand that, but in this case the only Julia + Ipopt installation working with additional linear solvers (HSL) is the one that was installed via homebrew. Installing Julia by downloading the official .dmg file (or juliaup) is the one that is causing the segfault, whereas the homebrew installation with HSL solvers works well after export DYLD_LIBRARY_PATH=/Users/henrique/coinbrew/Ipopt/dist/lib.

This is not true. You can use any linear solver with the default binaries that we provide by following https://github.com/jump-dev/Ipopt.jl#linear-solvers.

This says I should be able to use one of the following:

• Setting an environment variable export DL_LOAD_PATH=/full/path/somewhere/lib
  • I tried to set only export DYLD_LOAD_PATH=/Users/henrique/coinbrew/Ipopt/dist/lib and using Ipopt does not segfault, but it does not find the extra linear solvers either.
  • I tried setting both that and export DYLD_LIBRARY_PATH=/Users/henrique/coinbrew/Ipopt/dist/lib, but then it segfaults just the same on using Ipopt.
• Setting hsllib with set_optimizer_attribute(model, "hsllib","full/path/somewhere/lib/libhsl.dylib")
  • I am not using JuMP, so set_optimizer_attribute would not help.

I've also tried creating an alias to /Users/henrique/coinbrew/Ipopt/dist/lib/libcoinhsl.2.dylib as /Users/henrique/coinbrew/Ipopt/dist/lib/libhsl.dylib, but it didn't help.

Did you follow the instructions here: https://jump.dev/JuMP.jl/stable/developers/custom_solver_binaries/?
Setting DYLD_LIBRARY_PATH isn't sufficient to use your binaries. It's probably the cause of the segfault.

I gave Overriding an entire artifact a go just now, but I get the same segftault as soon as I try using Ipopt_jll:

julia> using Ipopt_jll
julia(9167,0x1fa835b40) malloc: *** error for object 0x117ae47d0: pointer being freed was not allocated
julia(9167,0x1fa835b40) malloc: *** set a breakpoint in malloc_error_break to debug

[9167] signal (6): Abort trap: 6
in expression starting at REPL[2]:1
__pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line)
Allocations: 2996 (Pool: 2985; Big: 11); GC: 0
[1]    9166 abort      julia --project

from ipopt.jl.

I'm confused. Are you trying to use the libipopt provided by coinbrew or just libhsl?

Is the the thrust of your issue that setting DYLD_LIBRARY_PATH to that coinbrew directory aborts the official binary of Julia, irrespective of whether you try to run Ipopt with HSL?

If so, my solution would be: don't use coinbrew to build HSL. Just download the official source code from the HSL website and compile manually on your machine. @amontoison is also working with the HSL folks to provide an official precompiled binaries for Julia that should be available very shortly.

The mess of trying to get different binaries to work together is why we're moving away from third party builds wherever possible. I don't know how the homebrew/coinbrew/official Julia builds all interact, but it isn't pretty and the list of things that could be going wrong is long.

I am not using JuMP, so set_optimizer_attribute would not help.

It's just the Ipopt option. hsllib is not JuMP-specific. You can use

from ipopt.jl.

Thank you for your replies, and apologies for only getting back to you now.

I tried what Oscar suggested above (i.e. compile everything from source instead of using coinbrew), but without success. At first, I got clang: error: unsupported option '-fopenmp', but that went away by setting CC=gcc-13. For completeness, here are the steps I tried:

1. Clone Ipopt and ThirdParty-HSL;
2. Download coinhsl-xxxx.xx.xx.zip from https://licences.stfc.ac.uk/product/coin-hsl;
3. Extract the coinhsl-xxxx.xx.xx.zip, rename it as just coinhsl, and place it inside the ThirdParty-HSL folder;
4. From the root of ThirdParty-HSL, run ./configure CC=gcc-13 CXX=g++-13, make, and sudo make install;
5. Create a build inside Ipopt, cd into it, run ../configure CC=gcc-13 CXX=g++-13, make, make test (everything passed), and finally sudo make install;
6. Add export DYLD_LIBRARY_PATH=/usr/local/lib to .bashrc/.zshrc.

However, if I do using Ipopt from a Julia REPL at this point, I am still getting a malloc: *** error for object 0x116a2c7d0: pointer being freed was not allocated.

I noticed that JuliaHSL is now available, so I will give it a try tomorrow.

from ipopt.jl.

You shouldn't need to compile Ipopt, only the HSL libraries.

I noticed that JuliaHSL is now available

👍 this should mean no more compilation problems. See https://github.com/jump-dev/Ipopt.jl#hsl

from ipopt.jl.

Did you try following the installation instructions in the coinhsl-archive download?

(base) oscar@Oscars-MBP coinhsl-archive-2023.05.26 % brew install meson
... stuff ...
(base) oscar@Oscars-MBP coinhsl-archive-2023.05.26 % mkdir build
(base) oscar@Oscars-MBP coinhsl-archive-2023.05.26 % meson setup builddir --buildtype=release --prefix=build
... stuff ...
(base) oscar@Oscars-MBP coinhsl-archive-2023.05.26 % meson setup builddir --buildtype=release --prefix=/Users/oscar/Downloads/coinhsl-archive-2023.05.26/build
... stuff ...
(base) oscar@Oscars-MBP coinhsl-archive-2023.05.26 % meson compile -C builddir
... stuff ...
(base) oscar@Oscars-MBP coinhsl-archive-2023.05.26 % julia --project=.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.7 (2022-07-19)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(coinhsl-archive-2023.05.26) pkg> add Ipopt JuMP
# ... stuff ...

(coinhsl-archive-2023.05.26) pkg> st
      Status `~/Downloads/coinhsl-archive-2023.05.26/Project.toml`
  [b6b21f68] Ipopt v1.4.0
  [4076af6c] JuMP v1.11.1

julia> using JuMP, Ipopt

julia> model = Model(Ipopt.Optimizer)
A JuMP Model
Feasibility problem with:
Variables: 0
Model mode: AUTOMATIC
CachingOptimizer state: EMPTY_OPTIMIZER
Solver name: Ipopt

julia> set_attribute(model, "linear_solver", "ma27")

julia> set_attribute(model, "hsllib", "/Users/oscar/Downloads/coinhsl-archive-2023.05.26/build/lib/libcoinhsl.dylib")

julia> @variable(model, x)
x

julia> @objective(model, Min, x^2)
x²

julia> optimize!(model)

******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
 Ipopt is released as open source code under the Eclipse Public License (EPL).
         For more information visit https://github.com/coin-or/Ipopt
******************************************************************************

This is Ipopt version 3.14.4, running with linear solver ma27.

Number of nonzeros in equality constraint Jacobian...:        0
Number of nonzeros in inequality constraint Jacobian.:        0
Number of nonzeros in Lagrangian Hessian.............:        1

Total number of variables............................:        1
                     variables with only lower bounds:        0
                variables with lower and upper bounds:        0
                     variables with only upper bounds:        0
Total number of equality constraints.................:        0
Total number of inequality constraints...............:        0
        inequality constraints with only lower bounds:        0
   inequality constraints with lower and upper bounds:        0
        inequality constraints with only upper bounds:        0

iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
   0  0.0000000e+00 0.00e+00 0.00e+00  -1.0 0.00e+00    -  0.00e+00 0.00e+00   0

Number of Iterations....: 0

                                   (scaled)                 (unscaled)
Objective...............:   0.0000000000000000e+00    0.0000000000000000e+00
Dual infeasibility......:   0.0000000000000000e+00    0.0000000000000000e+00
Constraint violation....:   0.0000000000000000e+00    0.0000000000000000e+00
Variable bound violation:   0.0000000000000000e+00    0.0000000000000000e+00
Complementarity.........:   0.0000000000000000e+00    0.0000000000000000e+00
Overall NLP error.......:   0.0000000000000000e+00    0.0000000000000000e+00


Number of objective function evaluations             = 1
Number of objective gradient evaluations             = 1
Number of equality constraint evaluations            = 0
Number of inequality constraint evaluations          = 0
Number of equality constraint Jacobian evaluations   = 0
Number of inequality constraint Jacobian evaluations = 0
Number of Lagrangian Hessian evaluations             = 0
Total seconds in IPOPT                               = 0.104

EXIT: Optimal Solution Found.

from ipopt.jl.

Thank you for your replies, and apologies for only getting back to you now.

I tried what Oscar suggested above (i.e. compile everything from source instead of using coinbrew), but without success. At first, I got clang: error: unsupported option '-fopenmp', but that went away by setting CC=gcc-13.

Where did you get this error?
Is it in the build system of ThirdParty-HSL of COIN-HSL?
We need to use -openmp is we compile Fortran / C / C++ code with the nagfor / clang / clang++ compilers.

I highly recommend to download the HSL_jll.jl archive available with JuliaHSL.
It should solve all issues.

from ipopt.jl.

Oh, I didn't realise that there was a new way for building the HSL routines! I tried to use meson and it worked really well on the first attempt. 🎉

The steps I took were very similar to the ones Oscar shared above:

1. Extract coinhsl-2023.05.26.zip
2. cd coinhsl-2023.05.26.zip
3. meson setup builddir --buildtype=release
4. meson compile -C builddir
5. meson install -C builddir
   Whose output looked like this:

   ninja: Entering directory `/Users/henrique/Downloads/coinhsl-2023.05.26/builddir'
   ninja: no work to do.
   Installing libcoinhsl.dylib to /opt/homebrew/lib
   Installing /Users/henrique/Downloads/coinhsl-2023.05.26/hsl_ma77/C/hsl_ma77s.h to /opt/homebrew/include
   Installing /Users/henrique/Downloads/coinhsl-2023.05.26/hsl_ma77/C/hsl_ma77d.h to /opt/homebrew/include
   Installing /Users/henrique/Downloads/coinhsl-2023.05.26/hsl_ma86/C/hsl_ma86s.h to /opt/homebrew/include
   Installing /Users/henrique/Downloads/coinhsl-2023.05.26/hsl_ma86/C/hsl_ma86d.h to /opt/homebrew/include
   Installing /Users/henrique/Downloads/coinhsl-2023.05.26/hsl_ma97/C/hsl_ma97s.h to /opt/homebrew/include
   Installing /Users/henrique/Downloads/coinhsl-2023.05.26/hsl_ma97/C/hsl_ma97d.h to /opt/homebrew/include
   Installing /Users/henrique/Downloads/coinhsl-2023.05.26/hsl_mc68/C/hsl_mc68i.h to /opt/homebrew/include
   

At this point, I was able to use the routines directly with Ipopt in one of two ways:

• By setting the path to hsllib as an Ipopt option with Ipopt.AddIpoptStrOption(prob, "hsllib", "/opt/homebrew/lib/libcoinhsl.dylib"); or
• By adding export DYLD_LIBRARY_PATH=/opt/homebrew/lib to the .bashrc/.zshrc.
  • Note that this one required also renaming libcoinhsl.dylib as libhsl.dylib.

Thank you for your help. It is nice to know that this more "manual" approach still works.

Anyway, I am curious to try JuliaHSL, so I will give that a go soon too.

from ipopt.jl.

Where did you get this error?
Is it in the build system of ThirdParty-HSL of COIN-HSL?
We need to use -openmp is we compile Fortran / C / C++ code with the nagfor / clang / clang++ compilers.

I got it when trying to build Ipopt from source with Apple clang 14 instead of gcc-13.

from ipopt.jl.

Nice to hear you got it sorted

from ipopt.jl.

Did you not need to set the hsllib attribute?

from ipopt.jl.

Do we also need to remove the quarantine with the archives JuliaHSL-2023.5.26.tar.gz and JuliaHSL-2023.5.26.zip?

Yes, otherwise we get the macOS pop-up that I have shared in my previous message, alongside the following error in the stdout of the Julia session:

julia> import HSL_jll
[ Info: Precompiling HSL_jll [017b0a0e-03f4-516a-9b91-836bbd1904dd]
ERROR: InitError: could not load library "/Users/henrique/Downloads/HSL_jll.jl-2023.5.26/override/lib/aarch64-apple-darwin-libgfortran5/libhsl.dylib"
dlopen(/Users/henrique/Downloads/HSL_jll.jl-2023.5.26/override/lib/aarch64-apple-darwin-libgfortran5/libhsl.dylib, 0x0001): tried: '/Users/henrique/knitro/lib/libhsl.dylib' (no such file), '/libhsl.dylib' (no such file), '/Users/henrique/Downloads/HSL_jll.jl-2023.5.26/override/lib/aarch64-apple-darwin-libgfortran5/libhsl.dylib' (code signature in <C8AD7334-EF5B-360F-B0E2-7FA92922192C> '/Users/henrique/Downloads/HSL_jll.jl-2023.5.26/override/lib/aarch64-apple-darwin-libgfortran5/libhsl.dylib' not valid for use in process: library load disallowed by system policy), '/System/Volumes/Preboot/Cryptexes/OS/Users/henrique/Downloads/HSL_jll.jl-2023.5.26/override/lib/aarch64-apple-darwin-libgfortran5/libhsl.dylib' (no such file), '/Users/henrique/Downloads/HSL_jll.jl-2023.5.26/override/lib/aarch64-apple-darwin-libgfortran5/libhsl.dylib' (code signature in <C8AD7334-EF5B-360F-B0E2-7FA92922192C> '/Users/henrique/Downloads/HSL_jll.jl-2023.5.26/override/lib/aarch64-apple-darwin-libgfortran5/libhsl.dylib' not valid for use in process: library load disallowed by system policy)
Stacktrace:
  [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
    @ Base.Libc.Libdl ./libdl.jl:117
  [2] dlopen(s::String, flags::UInt32)
    @ Base.Libc.Libdl ./libdl.jl:116
  [3] macro expansion
    @ ~/.julia/packages/JLLWrappers/QpMQW/src/products/library_generators.jl:54 [inlined]
  [4] __init__()
    @ HSL_jll ~/Downloads/HSL_jll.jl-2023.5.26/src/wrappers/aarch64-apple-darwin-libgfortran5.jl:12
  [5] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1074
  [6] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1020
  [7] _tryrequire_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String)
    @ Base ./loading.jl:1407
  [8] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1781
  [9] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1625
 [10] macro expansion
    @ ./loading.jl:1613 [inlined]
 [11] macro expansion
    @ ./lock.jl:267 [inlined]
 [12] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1576
during initialization of module HSL_jll

from ipopt.jl.

@ferrolho
I talked about the archives with the source code (JuliaHSL) and not the precompiled versions (HSL_jll.jl).
Because JuliaHSL doesn't contain any dylib file, I think that we don't need to remove any quarantine.

from ipopt.jl.

Did you not need to set the hsllib attribute?

That's right. For this approach (importing HSL_jll), I noticed there is no need to set the hsllib attribute.

However, for the other approach (building the HSL routines with Meson), I had to either set the hsllib attribute or to export the path to the folder where the compiled library was installed to. (This is what I was referring to at the end of my comment above #360 (comment).)

from ipopt.jl.

I talked about the archives with the source code (JuliaHSL) and not the precompiled versions (HSL_jll.jl).
Because JuliaHSL doesn't contain any dylib, I think that we don't need to remove any quarantine.

Oh, I see what you mean now. I did not try to build it from the source (JuliaHSL-2023.5.26.zip). I suspect it won't be needed, but I can try it out later today and let you know. Just to confirm, that source should be built with Meson too, right?

from ipopt.jl.

Thanks! I'm working on Linux so that's quite hard to test it.
Yes, the build system is based on Meson too.
Meson is perfect to do cross-compilation and it is (from my point of view) the best build system to generate HSL_jll.jl.

from ipopt.jl.