juliapackaging / binaryprovider.jl Goto Github PK
View Code? Open in Web Editor NEWA reliable binary provider for Julia
License: Other
A reliable binary provider for Julia
License: Other
We should detect that e.g. libgfortran.5.dylib
is loaded and freak out, refusing to download.
I was trying to build Pkg.build("Sundials")
when I got this error.
LoadError: Hash Mismatch! Expected sha256: f34ad4faa4d0c9841e15348a48c8ab5780af56da23af9d46274192c6bbcf1049 Calculated sha256: 4b5f9e8781909e722c51e134f68e83e6ca99ab612a5dab8ecc3e0f8057cfcdfd while loading /home/sohan/.julia/v0.6/Sundials/deps/build.jl, in expression starting on line 54
Can anyone tell what's causing this error ?
Note:- I'm behind a campus-proxy, but other julia packages get installed without a problem.
I'm on Ubuntu 16.04 LTS(Xenial amd64) using juila v0.6.0.
In RingBuffers.jl I add the RTLD_GLOBAL flag to my dlopen
call, so that C libraries from other packages (e.g. PortAudio.jl) can call the RingBuffer's C functions. BinaryProvider is definitely a slicker way to do what I was doing before, but it doesn't have a hook to override those flags.
Would a PR for that be accepted?
CodecZlib.jl requires libz: https://github.com/bicycle1885/CodecZlib.jl/blob/master/deps/build.jl
However, on FreeBSD it still downloads sources and breaks, even though /usr/lib/libz.so exists.
There seem to be several places that use Libdl.foo
without first importing the Libdl module, which is required on 0.7. (For 0.6 compatibility, import Compat.Libdl
). For example
The latter case is in generated code for deps.jl
so it triggers deprecation warnings in packages (see e.g. JuliaIO/Blosc.jl#56).
Suggest to include proxy support using default values on windows:
webclient_code = """
[System.Net.ServicePointManager]::SecurityProtocol =
[System.Net.SecurityProtocolType]::Tls12;
$webclient = (New-Object System.Net.Webclient);
$webclient.Headers.Add("user-agent", "$agent");
$webclient.Proxy.Credentials =[System.Net.CredentialCache]::DefaultNetworkCredentials;
$webclient.DownloadFile("$url", "$path")
"""
When building StringEncodings.jl, the log file shows a warning that says Could not extract the platform key
See details at JuliaStrings/StringEncodings.jl#31
When installing CodecZlib
(dependency decided by Pkg.update()
), I got the following error:
curl: (48) An unknown option was passed in to libcurl
Here is the full log when trying to build only this package:
INFO: Building CodecZlib
Info: Downloading https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.0/Zlib.x86_64-linux-gnu.tar.gz to /raid/home/thibaut/.julia/v0.6/CodecZlib/deps/usr/downloads/Zlib.x86_64-linux-gnu.tar.gz...
[16:28:38] curl: (48) An unknown option was passed in to libcurl
==========================================================================[ ERROR: CodecZlib ]==========================================================================
LoadError: Could not download https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.0/Zlib.x86_64-linux-gnu.tar.gz to /raid/home/thibaut/.julia/v0.6/CodecZlib/deps/usr/downloads/Zlib.x86_64-linux-gnu.tar.gz
while loading /raid/home/thibaut/.julia/v0.6/CodecZlib/deps/build.jl, in expression starting on line 92
========================================================================================================================================================================
============================================================================[ BUILD ERRORS ]============================================================================
WARNING: CodecZlib had build errors.
- packages with build errors remain installed in /raid/home/thibaut/.julia/v0.6
- build the package(s) and all dependencies with `Pkg.build("CodecZlib")`
- build a single package by running its `deps/build.jl` script
========================================================================================================================================================================
What's strange is that I have the latest version of curl installed:
curl 7.59.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.0.2n zlib/1.2.11
Release-Date: 2018-03-14
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy
I am using Julia 0.6.2:
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_) | Documentation: https://docs.julialang.org
_ _ _| |_ __ _ | Type "?help" for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.6.2 (2017-12-13 18:08 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org/ release
|__/ | x86_64-pc-linux-gnu
julia> versioninfo()
Julia Version 0.6.2
Commit d386e40c17 (2017-12-13 18:08 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
LAPACK: libopenblas64_
LIBM: libopenlibm
LLVM: libLLVM-3.9.1 (ORCJIT, ivybridge)
Hi all,
I'm unable to install Ipopt and need some assistance. When I run build Ipopt
(I get the same error with add Ipopt
and add Ipopt#master
), I get the following error on a fresh julia install:
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.0.1 (2018-09-29)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(v1.0) pkg> add Ipopt
Cloning default registries into /Users/$USER/.julia/registries
Cloning registry General from "https://github.com/JuliaRegistries/General.git"
Updating registry at `~/.julia/registries/General`
Updating git-repo `https://github.com/JuliaRegistries/General.git`
Resolving package versions...
Installed BinDeps ────────── v0.8.10
Installed Compat ─────────── v1.3.0
Installed Ipopt ──────────── v0.4.2
Installed MathProgBase ───── v0.7.7
Installed BinaryProvider ─── v0.5.2
Installed MathOptInterface ─ v0.6.1
Installed URIParser ──────── v0.4.0
Updating `~/.julia/environments/v1.0/Project.toml`
[b6b21f68] + Ipopt v0.4.2
Updating `~/.julia/environments/v1.0/Manifest.toml`
[9e28174c] + BinDeps v0.8.10
[b99e7846] + BinaryProvider v0.5.2
[34da2185] + Compat v1.3.0
[b6b21f68] + Ipopt v0.4.2
[b8f27783] + MathOptInterface v0.6.1
[fdba3010] + MathProgBase v0.7.7
[30578b45] + URIParser v0.4.0
[2a0f44e3] + Base64
[ade2ca70] + Dates
[8bb1440f] + DelimitedFiles
[8ba89e20] + Distributed
[b77e0a4c] + InteractiveUtils
[76f85450] + LibGit2
[8f399da3] + Libdl
[37e2e46d] + LinearAlgebra
[56ddb016] + Logging
[d6f4376e] + Markdown
[a63ad114] + Mmap
[44cfe95a] + Pkg
[de0858da] + Printf
[3fa0cd96] + REPL
[9a3f8284] + Random
[ea8e919c] + SHA
[9e88b42a] + Serialization
[1a1011a3] + SharedArrays
[6462fe0b] + Sockets
[2f01184e] + SparseArrays
[10745b16] + Statistics
[8dfed614] + Test
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
Building Ipopt → `~/.julia/packages/Ipopt/5a2PI/deps/build.log`
┌ Error: Error building `Ipopt`:
│ ┌ Warning: platform_key() is deprecated, use platform_key_abi() from now on
│ │ caller = ip:0x0
│ └ @ Core :-1
│ ┌ Warning: Could not extract the platform key of https://github.com/staticfloat/IpoptBuilder/releases/download/v3.12.8-9/Ipopt.x86_64-apple-darwin14.tar.gz; continuing...
│ └ @ BinaryProvider ~/.julia/packages/BinaryProvider/1nGWd/src/Prefix.jl:185
│ [ Info: Downloading https://github.com/staticfloat/IpoptBuilder/releases/download/v3.12.8-9/Ipopt.x86_64-apple-darwin14.tar.gz to /Users/$USER/.julia/packages/Ipopt/5a2PI/deps/usr/downloads/Ipopt.x86_64-apple-darwin14.tar.gz...
│ ERROR: LoadError: LibraryProduct(nothing, ["libipopt"], :libipopt, "Prefix(/Users/$USER/.julia/packages/Ipopt/5a2PI/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│ [1] error(::String) at ./error.jl:33
│ [2] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{Product,1}) at /Users/$USER/.julia/packages/BinaryProvider/1nGWd/src/Products.jl:408
│ [3] write_deps_file(::String, ::Array{Product,1}) at /Users/$USER/.julia/packages/BinaryProvider/1nGWd/src/Products.jl:395
│ [4] top-level scope at /Users/$USER/.julia/packages/Ipopt/5a2PI/deps/build.jl:51
│ [5] include at ./boot.jl:317 [inlined]
│ [6] include_relative(::Module, ::String) at ./loading.jl:1041
│ [7] include(::Module, ::String) at ./sysimg.jl:29
│ [8] include(::String) at ./client.jl:388
│ [9] top-level scope at none:0
│ in expression starting at /Users/$USER/.julia/packages/Ipopt/5a2PI/deps/build.jl:13
[12:32:07] ######################################################################## 100.0%
└ @ Pkg.Operations /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1069
(v1.0) pkg>
Question 1: What is /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1069
? I don't have a osx
user on my machine. Where is this hardcoded message coming from? The Julia base image?
This tar.gz file that is being downloaded in this process is located here.
If I cd
into the Ipopt folder and run otool
on the libraries, I see the following:
otool -L ~/.julia/packages/Ipopt/5a2PI/deps/usr/lib/libipopt.dylib
/Users/$USER/.julia/packages/Ipopt/5a2PI/deps/usr/lib/libipopt.dylib:
/workspace/destdir/lib/libipopt.1.dylib (compatibility version 12.0.0, current version 12.8.0)
@loader_path/libcoinmumps.1.dylib (compatibility version 8.0.0, current version 8.0.0)
@rpath/libgfortran.4.dylib (compatibility version 5.0.0, current version 5.0.0)
@rpath/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
/opt/x86_64-apple-darwin14/x86_64-apple-darwin14/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.24.0)
Question 2: Why is there a /opt/x86_64-apple-darwin14/x86_64-apple-darwin14/lib/libstdc++.6.dylib
? Shouldn't this also use @rpath
? Or at least point to /usr/lib/libc++.1.dylib
instead?
This is what happens when I run julia
and try to load the library.
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.0.1 (2018-09-29)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> import Libdl
julia> Libdl.dlopen("./libipopt.dylib")
ERROR: could not load library "./libipopt.dylib"
dlopen(./libipopt.dylib, 1): Library not loaded: /opt/x86_64-apple-darwin14/x86_64-apple-darwin14/lib/libstdc++.6.dylib
Referenced from: /Users/$USER/.julia/packages/Ipopt/5a2PI/deps/usr/lib/libipopt.1.10.8.dylib
Reason: image not found
Stacktrace:
[1] dlopen(::String, ::UInt32) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Libdl/src/Libdl.jl:97 (repeats 2 times)
[2] top-level scope at none:0
julia>
Any thoughts on how to fix this issue? It appears to work on other colleagues' Mac machines.
Better solution than living with deprecation warnings
When putting together install/build scripts, it's a bit inconvenient and not entirely clear how to get the correct hash. After digging around in BinaryProvider I came up with the following function to get a hash:
"""
url2hash(url::String)
usage:
hash = url2hash("https://downloads.xiph.org/releases/vorbis/libvorbis-1.3.6.tar.xz")
"""
function url2hash(url::String)
path = download(url)
open(io-> bytes2hex(BinaryProvider.sha256(io)), path)
end
Maybe we can add this somewhere to BinaryProvider?
┌ Warning: `warn()` is deprecated, use `@warn` instead.
│ caller = #warn#872(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::String, ::Vararg{String,N} where N) at deprecated.jl:1010
└ @ Base deprecated.jl:1010
WARNING: Could not spawn `curl -C - '-#' -f -o /home/travis/.julia/v0.7/MbedTLS/deps/usr/downloads/MbedTLS.x86_64-linux-gnu.tar.gz -L https://github.com/quinnj/MbedTLSBuilder/releases/download/v0.5/MbedTLS.x86_64-linux-gnu.tar.gz`
┌ Error: ------------------------------------------------------------
│ # Build failed for MbedTLS
│ exception =
│ LoadError: MethodError: no method matching spawn(::Cmd, ::Tuple{Base.DevNullStream,Pipe,Pipe})
│ Closest candidates are:
│ spawn(::Base.AbstractCmd) at deprecated.jl:55
│ Stacktrace:
│ [1] #OutputCollector#5(::Bool, ::Bool, ::Base.TTY, ::Type, ::Cmd) at /home/travis/.julia/v0.7/BinaryProvider/src/OutputCollector.jl:115
│ [2] (::getfield(Core, Symbol("#kw#Type")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::Type{BinaryProvider.OutputCollector}, ::Cmd) at ./<missing>:0
│ [3] #download#89(::Bool, ::Function, ::String, ::String) at /home/travis/.julia/v0.7/BinaryProvider/src/PlatformEngines.jl:421
from this log: https://travis-ci.org/JuliaWeb/MbedTLS.jl/jobs/348374182
It might be nice to provide an easy way to rename all the symbols in a library, to avoid conflicts with other external libraries like we encountered in JuliaLang/julia#4923.
For example, if you included GNU objcopy
in your cross-compile setup, then something like --prefix-symbols
could be used for this without modifying any source code.
Currently, BinaryProvider assumes that .dll
files are installed into the bin
directory. If they are installed into lib
, I get an error like this:
LoadError: BinaryProvider.LibraryProduct("C:\\Users\\appveyor\\.julia\\v0.6\\Blosc\\deps\\usr\\bin", String["libblosc"], :libblosc, "Prefix(C:\\Users\\appveyor\\.julia\\v0.6\\Blosc\\deps\\usr)") is not satisfied, cannot generate deps.jl!
while loading C:\Users\appveyor\.julia\v0.6\Blosc\deps\build.jl, in expression starting on line 40
As a workaround, I notice that e.g. MbedTLSBuilder's build_tarballs.jl script has a hack to copy .dll
files to the bin
directory:
if [ $target == "x86_64-w64-mingw32" ]; then
cp $prefix/lib/*.dll $prefix/bin/.
elif [ $target == "i686-w64-mingw32" ]; then
cp $prefix/lib/*.dll $prefix/bin/.
Couldn't BinaryProvider look in both the lib
and bin
directories for .dll
files on Windows?
(Or alternatively BinaryBuilder's build_tarballs
function could move .dll
files into the bin
directory.)
It's pretty common for Unixy makefiles to install all libraries into lib
, even when cross-compiling, and it seems unfortunate to require every build_tarballs.jl
to include some boilerplate code to move the .dll
files to a different directory.
I've got a working BinaryBuilder release for my package here: https://github.com/BenJWard/BSGBuilder/releases/tag/v0.6.0
When I use the build.jl file in a package on my local Mojave macbook pro, the following happens:
(v1.0) pkg> activate .
(BSG) pkg> add BinaryProvider
Updating registry at `~/.julia/registries/General`
Updating git-repo `https://github.com/JuliaRegistries/General.git`
Resolving package versions...
Updating `~/Desktop/BSG/Project.toml`
[b99e7846] + BinaryProvider v0.5.2
Updating `~/Desktop/BSG/Manifest.toml`
[no changes]
(BSG) pkg> build
Building CxxWrap → `~/.julia/packages/CxxWrap/KcmSi/deps/build.log`
Building BSG ────→ `~/Desktop/BSG/deps/build.log`
Resolving package versions...
┌ Error: Error building `BSG`:
│ ERROR: LoadError: ArgumentError: Unable to find matching download for MacOS(:x86_64, compiler_abi=CompilerABI(:gcc8, :cxx03))
│ Stacktrace:
│ [1] choose_download(::Dict{Platform,Tuple{String,String}}, ::MacOS) at /Users/bward/.julia/packages/BinaryProvider/1nGWd/src/PlatformNames.jl:733
│ [2] top-level scope at none:0
│ [3] include at ./boot.jl:317 [inlined]
│ [4] include_relative(::Module, ::String) at ./loading.jl:1041
│ [5] include(::Module, ::String) at ./sysimg.jl:29
│ [6] include(::String) at ./client.jl:388
│ [7] top-level scope at none:0
│ in expression starting at /Users/bward/Desktop/BSG/deps/build.jl:24
└ @ Pkg.Operations ~/github/julia/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1069
My binary releases only use cxx11 and either gcc8 or gcc7 as their compiler_abi options, because bsg really needs cxx11 as a minimum. Is there a way to change the cxx standard options when downloading a build?
To reproduce, Pkg.add("SpecialFunctions")
then take a look at the generated deps.jl
file. Mine looks like
## This file autogenerated by BinaryProvider.write_deps_file().
## Do not edit.
##
## Include this file within your main top-level source, and call
## `check_deps()` from within your module's `__init__()` method
if isdefined((@static VERSION < v"0.7.0-DEV.484" ? current_module() : @__MODULE__), :Compat)
import Compat.Libdl
elseif VERSION >= v"0.7.0-DEV.3382"
import Libdl
end
const openspecfun = joinpath(dirname(@__FILE__), "usr/lib/libopenspecfun.1.3.dylib")
function check_deps()
global openspecfun
if !isfile(openspecfun)
error("$(openspecfun) does not exist, Please re-run Pkg.build(\"KvXoO\"), and restart Julia.")
end
if Libdl.dlopen_e(openspecfun) in (C_NULL, nothing)
error("$(openspecfun) cannot be opened, Please re-run Pkg.build(\"KvXoO\"), and restart Julia.")
end
end
Note in particular
Please re-run
Pkg.build(\"KvXoO\")
This is due to this line:
BinaryProvider.jl/src/Products.jl
Line 398 in d0e7869
/Users/alex/.julia/packages/SpecialFunctions/KvXoO/deps/deps.jl
.JuliaLang/Pkg.jl#25 attempts to implement asynchronous downloading and unpacking of tarballs for getting the packages.
There was some problems on Windows that resulted in a deadlock
I have made a branch here (with some debug output) https://github.com/KristofferC/Pkg3.jl/tree/kc/winrepro that for me reproduces the problem when running
julia> import Pkg3
<]> # enter pkg repl
pkg> install GLVisualize
.....
......
****************
Packages left to handle:
IdentityRanges scheduled: true
INFO: Installed CoordinateTransformations ─ v0.4.1
[DEADLOCK]
What the debug output is saying is that we have scheduled the package "IdentityRanges" to be downloaded and unpacked here: https://github.com/KristofferC/Pkg3.jl/blob/kc/winrepro/src/Operations.jl#L379 but the put!
statement on the next line is never reached. In the function install
(https://github.com/KristofferC/Pkg3.jl/blob/kc/winrepro/src/Operations.jl#L248) the acrhive is downloaded and uncompressed. Further debugging suggested that we are stuck waiting for the OutputCollector
task to finish and it never does, which is why the blocking take!
call at https://github.com/KristofferC/Pkg3.jl/blob/kc/winrepro/src/Operations.jl#L395 never finishes.
Moving to just downloading and unpacking using the generated commands from BinaryProvider worked around the issue, so perhaps there is some problem with the OutputCollector.
I am sorry to not be able to provide a more minimum example. The issue is quite finicky since depending on where I put print statements the pkg command might succeed or not.
FWIW, It seems that it is usually the first task that gets scheduled that gets stuck, if that can help debugging...
Perhaps this could be considered as using internals of the package, but anyway:
When Interrupting a download
with e.g. ^C I get:
^CERROR (unhandled task failure): InterruptException:
Stacktrace:
[1] wait() at /Applications/Julia-0.6.app/Contents/Resources/julia/lib/julia/sys.dylib:? (repeats 3 times)
[2] macro expansion at .../BinaryProvider/src/OutputCollector.jl:59 [inlined]
[3] (::BinaryProvider.##2#4{Condition,Task})() at ./task.jl:335
and the call to download
then throws a normal ErrorException
that the download failed. In a situation where you try to download a file from different urls, you probably want to respond differently to a failure to download a file, and a ^C from the user.
Would it be possible to "handle the task failure" and propagate the InterruptException
out to the caller of download
?
via @ararslan:
"""
isexecutable(program)
Determine whether the given program name or path is executable using the current user's
permissions. This is roughly equivalent to querying `which program` at the command line
and checking that a result is found, but no shelling out occurs.
"""
function isexecutable(k::Symbol, prog::String)
X_OK = 1 << 0 # Taken from unistd.h
# If prog has a slash, we know the user wants to determine whether the given
# file exists and is executable
if '/' in prog || (iswindows(k) && '\\' in prog) # Windows can use / too
isfile(prog) || return false
return ccall(:access, Cint, (Ptr{UInt8}, Cint), prog, X_OK) == 0
end
path = get(ENV, "PATH", "")
# Something is definitely wrong if the user's path is empty...
@assert !isempty(path)
sep = iswindows(k) ? ';' : ':'
for dir in split(path, sep), file in readdir(dir)
if file == prog || (iswindows(k) && file == prog * ".exe")
p = joinpath(dir, file)
@assert isfile(p)
return ccall(:access, Cint, (Ptr{UInt8}, Cint), p, X_OK) == 0
end
end
false
end
isexecutable(prog::String) = isexecutable(Sys.KERNEL, prog)
When called from write_deps_file()
, locate()
will happily match any library that begins with the same string as the library we are looking for:
BinaryProvider.jl/src/Products.jl
Line 158 in cb403e6
In my case I was trying to use the library libgsl
, which existed in the same folder as the library libgslcblas
:
Info: Found a valid dl path libgslcblas.so while looking for libgsl
Info: /home/ludvigak/.julia/dev/GSL/deps/usr/lib/libgslcblas.so matches our search criteria of libgsl
I could fix this by changing
LibraryProduct(prefix, ["libgsl"], :libgsl)
to
LibraryProduct(prefix, ["libgsl."], :libgsl)
in the build script, but that was not obvious from the start.
This greedy matching could be considered a feature or a bug, depending on how you want things to work.
Ran into an issue today trying to deploy an app where my julia image was built in our build pipeline (jenkins), including building/precompiling packages like MbedTLS.jl, but then when pushed to the actual deploy server fails because the deps.jl
file for MbedTLS.jl was hard-coded w/ the absolute path from the build node. Could we just use relative paths in the generated deps.jl file so we don't run into these issues? Currently I'm having to "rebuild" on each node.
When downloading large files, (especially over intermittent connections) it should be possible to use curl's "resume" functionality to continue interrupted downloads.
I'm not sure this is a bug in this package, I opened a thread on discourse yesterday but got no reply. What I can't understand is why it works with Julia 0.7, but not with 0.6. Thus I suspect it might be a bug (or an error on my side that I can't see). In addition, I don't have a Windows box to play with, so it's a bit hard for me to track down the issue.
Below my issue report.
I'm trying to switch a package to BinaryBuilder.jl
. It works fine on most systems (at least, those that I can test with Travis), but I'm facing some difficulties with the Windows build: the library is successfully installed in Julia 0.7, but fails with Julia 0.6 with the following error:
LoadError: BinaryProvider.LibraryProduct(nothing, String["libwcs"], :libwcs, "Prefix(C:\\Users\\appveyor\\.julia\\v0.6\\WCS\\deps\\usr)") is not satisfied, cannot generate deps.jl!
while loading C:\Users\appveyor\.julia\v0.6\WCS\deps\build.jl, in expression starting on line 46
Here are some information:
build.jl
script in the Julia wrapper: https://github.com/giordano/WCS.jl/blob/b570c9004682ff6fd71bb1f0728416091cf3f7ca/deps/build.jlHas anyone a clue about why Julia 0.6 fails to install the library on Windows? Am I doing something wrong, maybe in the build script?
I'm not sure if this is a problem with the conda-forge julia package or this package, but using a package that requires BinaryProvider.jl with Anaconda fails to find the libLLVM symbols.
On a Linux system with docker, this can be reproduced by running:
docker run continuumio/anaconda /bin/bash -c "/opt/conda/bin/conda install -c conda-forge -yq julia && julia -e 'using Pkg; Pkg.add(\"ZMQ\")'"
This results in the following error:
Building ZMQ → `~/.julia/packages/ZMQ/ABGOx/deps/build.log`
┌ Error: Error building `ZMQ`:
│ ERROR: LoadError: LoadError: Unable to open libLLVM!
│ Stacktrace:
│ [1] error(::String) at ./error.jl:33
│ [2] (::getfield(BinaryProvider, Symbol("#open_libllvm#114")))() at /root/.julia/packages/BinaryProvider/cVlaj/src/PlatformNames.jl:649
│ [3] detect_cxx11_string_abi() at /root/.julia/packages/BinaryProvider/cVlaj/src/PlatformNames.jl:652
│ [4] detect_compiler_abi() at /root/.julia/packages/BinaryProvider/cVlaj/src/PlatformNames.jl:665
│ [5] top-level scope at none:0
│ [6] include at ./boot.jl:317 [inlined]
│ [7] include_relative(::Module, ::String) at ./loading.jl:1038
│ [8] include at ./sysimg.jl:29 [inlined]
│ [9] include(::String) at /root/.julia/packages/BinaryProvider/cVlaj/src/BinaryProvider.jl:1
│ [10] top-level scope at none:0
│ [11] include at ./boot.jl:317 [inlined]
│ [12] include_relative(::Module, ::String) at ./loading.jl:1038
│ [13] include(::Module, ::String) at ./sysimg.jl:29
│ [14] top-level scope at none:2
│ [15] eval at ./boot.jl:319 [inlined]
│ [16] eval(::Expr) at ./client.jl:389
│ [17] top-level scope at ./none:3
│ in expression starting at /root/.julia/packages/BinaryProvider/cVlaj/src/PlatformNames.jl:682
│ in expression starting at /root/.julia/packages/BinaryProvider/cVlaj/src/BinaryProvider.jl:12
│ ERROR: LoadError: Failed to precompile BinaryProvider [b99e7846-7c00-51b0-8f62-c81ae34c0232] to /root/.julia/compiled/v1.0/BinaryProvider/ek6VZ.ji.
│ Stacktrace:
│ [1] error(::String) at ./error.jl:33
│ [2] macro expansion at ./logging.jl:313 [inlined]
│ [3] compilecache(::Base.PkgId, ::String) at ./loading.jl:1184
│ [4] _require(::Base.PkgId) at ./logging.jl:311
│ [5] require(::Base.PkgId) at ./loading.jl:852
│ [6] macro expansion at ./logging.jl:311 [inlined]
│ [7] require(::Module, ::Symbol) at ./loading.jl:834
│ [8] include at ./boot.jl:317 [inlined]
│ [9] include_relative(::Module, ::String) at ./loading.jl:1038
│ [10] include(::Module, ::String) at ./sysimg.jl:29
│ [11] include(::String) at ./client.jl:388
│ [12] top-level scope at none:0
│ in expression starting at /root/.julia/packages/ZMQ/ABGOx/deps/build.jl:1
└ @ Pkg.Operations /home/conda/feedstock_root/build_artifacts/julia_1534540929950/work/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1068
This was not the case for Julia 0.6.1, although that used BinaryProvider v0.3.3
When verifying downloads, save hashes to a $(file).sha256
file, then just check mtime
timestamps to ensure that a large file hasn't changed since the last time we verified, so that we aren't constantly verifying multi-GB rootfs images in BinaryBuilder.
Please
Currently, BinaryProvider downloads into MyPackage/deps/downloads
and installs into MyPackage/deps/usr
. With Julia 1.0, every version of MyPackage installs into its own directory, which means that every update installs a new copy of the binary dependencies even if they didn't change.
I don't think this is acceptable in the long run. Proposal:
BinaryProvider should install binary dependencies into a global SHA-addressed directory, e.g. ~/.julia/BinaryProvider/{MyPackage-UUID}/sha
, so that they are shared between versions.
Some kind of hook should be added to Pkg.gc()
to allow binary dependencies that are no longer needed to be deleted.
Came across this in GEOSBuilder, building with TARGET=arm-linux-gnueabihf
.
https://travis-ci.org/JuliaGeo/GEOSBuilder/jobs/360575777#L507-L516
julia> triplet(BinaryProvider.Linux(:armv7l))
"arm-linux-gnueabihf"
julia> platform_key("arm-linux-gnueabihf")
WARNING: Platform `arm-linux-gnueabihf` is not an officially supported platform
BinaryProvider.UnknownPlatform()
julia> platform_key("armv7l-linux-gnueabihf")
BinaryProvider.Linux(:armv7l, :glibc, :eabihf)
This can be reversed by chaning the dict in platform_key to :armv7l => "arm"
, but I'm not sure if this is the way to go, or that we need do deal with sub-arch explicitly.
Downstream issue: SciML/Sundials.jl#172 . CC @MartinOtter.
under Windows 7 and 10 creating symbolic links requires administrator rights (that I do not have). Would it be possible to change the setup-procedures so that no symbolic links are set?
When I edit build.jl so that it finds libxml2 from "/usr/lib/" and run it to generate deps.jl, it emits too many log messages that are not so informative (I know the directory includes such libraries in it):
prefix = Prefix(!isempty(ARGS) ? ARGS[1] : joinpath(@__DIR__, "usr"))
↓
prefix = Prefix("/usr")
~/w/EzXML (master|✚1…) $ julia-dev deps/build.jl
┌ Warning: `ismatch(r::Regex, s::AbstractString)` is deprecated, use `contains(s, r)` instead.
│ caller = valid_dl_path(::String, ::MacOS) at PlatformNames.jl:204
└ @ BinaryProvider PlatformNames.jl:204
[ Info: Found a valid dl path PN548_API.dylib while looking for libxml2
[ Info: Found a valid dl path PN548_HAL_OSX.dylib while looking for libxml2
[ Info: Found a valid dl path PN548_OSX.dylib while looking for libxml2
[ Info: Found a valid dl path libATCommandStudioDynamic.dylib while looking for libxml2
[ Info: Found a valid dl path libAccountPolicyTranslation.dylib while looking for libxml2
[ Info: Found a valid dl path libBSDPClient.A.dylib while looking for libxml2
[ Info: Found a valid dl path libBSDPClient.dylib while looking for libxml2
[ Info: Found a valid dl path libCRFSuite.dylib while looking for libxml2
[ Info: Found a valid dl path libCRFSuite0.12.dylib while looking for libxml2
[ Info: Found a valid dl path libChineseTokenizer.dylib while looking for libxml2
[ Info: Found a valid dl path libCoreStorage.dylib while looking for libxml2
᠁
more than 200 lines of messages here
᠁
[ Info: Found a valid dl path libxml2.2.dylib while looking for libxml2
[ Info: /usr/lib/libxml2.2.dylib matches our search criteria of libxml2
WARNING: importing deprecated binding Base.Libdl into Sys.
WARNING: Base.Libdl is deprecated, run `using Libdl` instead
likely near /Users/kenta/workspace/EzXML/deps/build.jl:26
I think these messages are astonishing but not so useful, so it should be less noisy.
Multiple reports which seem to be this same issue that users from the old Sundials on MacOS get both binaries when it updates, and that confuses BinaryProvider.jl
This is speculative as I am not an expert here.
On my Win10 unpack
fails because creation of symbolic links throws an error that is not caught.
Maybe on Win10 ENV["BINARYPROVIDER_COPYDEREF"]
should be set to "true"
by default on this platform as this solves the problem.
I am not sure if this is my local problem or a general problem for Win10.
(Related to issues #32 and #57)
My suggestion is to have the build.jl script, here we're using CodecXz as an example, somehow check whether the library it's looking for already exists on the system, and if so, proceed directly to creating the deps.jl file. In this specific case, FreeBSD has /usr/lib/liblzma.so.
I made CodecXz work by manually creating a deps.jl file with the right path to the library.
On a Win 7 machine:
julia> using BinaryProvider
┌ Warning: On Windows, creating file symlinks requires Administrator privileges
└ @ Base.Filesystem file.jl:789
julia> BinaryProvider.download("https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.2/Zlib.v1.2.11.x86_64-
w64-mingw32.tar.gz","zlib.tar.gz")
Exception setting "SecurityProtocol": "Cannot convert null to type "System.Net.SecurityProtocolType" due to invalid enu
meration values. Specify one of the following enumeration values and try again. The possible enumeration values are "Ss
l3, Tls"."
At line:1 char:35
+ [System.Net.ServicePointManager]:: <<<< SecurityProtocol =
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : PropertyAssignmentException
Exception calling "DownloadFile" with "2" argument(s): "The underlying connection was closed: An unexpected error occur
red on a send."
At line:5 char:24
+ $webclient.DownloadFile <<<< ("https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.2/Zlib.v1.2.11.x86_6
4-w64-mingw32.tar.gz", "zlib.tar.gz")
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : DotNetMethodException
ERROR: Could not download https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.2/Zlib.v1.2.11.x86_64-w64-mi
ngw32.tar.gz to zlib.tar.gz:
ErrorException("")
Stacktrace:
[1] error(::String) at .\error.jl:33
[2] #download#93(::Bool, ::Function, ::String, ::String) at .\logging.jl:313
[3] download(::String, ::String) at C:\Users\sbyrne\.julia\packages\BinaryProvider\cVlaj\src\PlatformEngines.jl:485
[4] top-level scope at none:0
This gives confusing errors, e.g. JuliaIO/CodecZlib.jl#24 (comment)
Base.download checks for this:
julia> download("https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.2/Zlib.v1.2.11.x86_64-w64-mingw32.tar
.gz")
Cannot start Windows PowerShell version 3 because it is not correctly installed.
ERROR: Downloading files requires Windows Management Framework 3.0 or later.
Stacktrace:
[1] error(::String) at .\error.jl:33
[2] download(::String, ::String) at .\download.jl:20
[3] download(::String) at .\download.jl:59
[4] top-level scope at none:
Currently, the build.jl
logic does nothing if the satisfied
returns true
for all the build products, i.e. if the libraries are all found in deps/usr
. This makes it impossible to update the binaries without either manually doing Pkg.rm(Pkg.dir("deps", "usr"), recursive=true)
or renaming the build products.
This seems like it makes it very hard to push fixes in the binaries to users.
Proposal: Pkg.build
should always re-download if the hash changes. In particular the build.jl
logic could be changed to something like:
unsatisfied = any(!satisfied(p; verbose=verbose) for p in products)
if haskey(download_info, platform_key())
url, tarball_hash = download_info[platform_key()]
if unsatisfied || !isinstalled(tarball_hash)
install(url, tarball_hash; prefix=prefix, force=true, verbose=verbose)
end
elseif unsatisfied
error("Your platform $(triplet(platform_key())) is not supported by this package!")
end
where BinaryProvider.isinstalled(tarball_hash)
checks some file that saves the hash.
Error in tests on 0.7 on win64 (binary download) in runtest.jl,L101.
100 # Next, test reading the output of a pipeline()
101 grepline = pipeline(sh(`-c 'printf "Hello\nWorld\nJulia"'`), `grep ul`)
102 oc = OutputCollector(grepline)
The problem is, the grep ul
command is not "wrapped" with busybox but windows tries to find some native "grep.exe" that doesn't generally exist (of course you might be lucky and windows can find some somewhere on path, e.g. from the git install - this is why the test could pass on travis or locally).
Full stacktrace here:
[ Info: Probing C:\Julia-0.7.0\bin\busybox.exe as a possibility...
[ Info: Probe successful for C:\Julia-0.7.0\bin\busybox.exe
[ Info: Found sh engine C:\Julia-0.7.0\bin\busybox.exe
1
2
┌ Warning: Could not spawn pipeline(`'C:\Julia-0.7.0\bin\busybox.exe' sh -c -c 'printf "Hello\nWorld\nJulia"'`, stdout=`grep ul`)
└ @ BinaryProvider C:\Users\phlavenk\.julia\packages\BinaryProvider\ZbFxn\src\OutputCollector.jl:120
OutputCollector: Error During Test at C:\Users\phlavenk\.julia\packages\BinaryProvider\ZbFxn\test\runtests.jl:33
Got exception Base.IOError("could not spawn `grep ul`: no such file or directory (ENOENT)", -4058) outside of a @test
IOError: could not spawn `grep ul`: no such file or directory (ENOENT)
Stacktrace:
[1] _jl_spawn(::String, ::Array{String,1}, ::Cmd, ::Tuple{Base.Libc.WindowsRawSocket,Base.PipeEndpoint,Base.PipeEndpoint}) at .\process.jl:370
[2] (::getfield(Base, Symbol("##499#500")){Cmd})(::Tuple{Base.Libc.WindowsRawSocket,Base.PipeEndpoint,Base.PipeEndpoint}) at .\process.jl:512
[3] setup_stdio(::getfield(Base, Symbol("##499#500")){Cmd}, ::Tuple{Base.Libc.WindowsRawSocket,Pipe,Pipe}) at .\process.jl:493
[4] #_spawn#498(::Base.ProcessChain, ::Function, ::Cmd, ::Tuple{Base.Libc.WindowsRawSocket,Pipe,Pipe}) at .\process.jl:511
[5] (::getfield(Base, Symbol("#kw##_spawn")))(::NamedTuple{(:chain,),Tuple{Base.ProcessChain}}, ::typeof(Base._spawn), ::Cmd, ::Tuple{Base.Libc.WindowsRawSocket,Pipe,Pipe}) at .\none:0
[6] #_spawn#496(::Nothing, ::Function, ::Base.OrCmds, ::Tuple{Base.DevNullStream,Pipe,Pipe}) at .\process.jl:415
[7] (::getfield(Base, Symbol("#kw##_spawn")))(::NamedTuple{(:chain,),Tuple{Nothing}}, ::typeof(Base._spawn), ::Base.OrCmds, ::Tuple{Base.DevNullStream,Pipe,Pipe}) at .\none:0
[8] #_spawn#495(::Nothing, ::Function, ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Pipe,Pipe}) at .\process.jl:401
[9] (::getfield(Base, Symbol("#kw##_spawn")))(::NamedTuple{(:chain,),Tuple{Nothing}}, ::typeof(Base._spawn), ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Pipe,Pipe}) at .\none:0
[10] #_spawn#495(::Nothing, ::Function, ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Base.DevNullStream,Pipe}) at .\process.jl:401
[11] (::getfield(Base, Symbol("#kw##_spawn")))(::NamedTuple{(:chain,),Tuple{Nothing}}, ::typeof(Base._spawn), ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Base.DevNullStream,Pipe}) at .\none:0
[12] #_spawn#495(::Nothing, ::Function, ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Base.DevNullStream,Base.DevNullStream}) at .\process.jl:401
[13] _spawn at .\process.jl:401 [inlined]
[14] #run#509(::Bool, ::Function, ::Base.CmdRedirect) at .\process.jl:672
[15] #run at .\none:0 [inlined]
[16] #OutputCollector#5(::Bool, ::Bool, ::Base.TTY, ::Type, ::Base.OrCmds) at C:\Users\phlavenk\.julia\packages\BinaryProvider\ZbFxn\src\OutputCollector.jl:118
[17] OutputCollector(::Base.OrCmds) at C:\Users\phlavenk\.julia\packages\BinaryProvider\ZbFxn\src\OutputCollector.jl:115
[18] macro expansion at C:\Users\phlavenk\.julia\packages\BinaryProvider\ZbFxn\test\runtests.jl:101 [inlined]
[19] macro expansion at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v0.7\Test\src\Test.jl:1079 [inlined]
[20] top-level scope at C:\Users\phlavenk\.julia\packages\BinaryProvider\ZbFxn\test\runtests.jl:34
[21] include at .\boot.jl:317 [inlined]
[22] include_relative(::Module, ::String) at .\loading.jl:1038
[23] include(::Module, ::String) at .\sysimg.jl:29
[24] include(::String) at .\client.jl:398
[25] top-level scope at none:0
[26] eval(::Module, ::Any) at .\boot.jl:319
[27] macro expansion at .\logging.jl:317 [inlined]
[28] exec_options(::Base.JLOptions) at .\client.jl:229
[29] _start() at .\client.jl:432
Test Summary: | Pass Error Total
OutputCollector | 15 1 16
ERROR: LoadError: Some tests did not pass: 15 passed, 0 failed, 1 errored, 0 broken.
in expression starting at C:\Users\phlavenk\.julia\packages\BinaryProvider\ZbFxn\test\runtests.jl:33
ERROR: Package BinaryProvider errored during testing
Example error:
INFO: Building Sundials
==============================[ ERROR: Sundials ]===============================
LoadError: InitError: No sh engines found. We looked for: C:\Users\Chris\AppData\Local\JuliaPro-0.6.2.2\Julia-0.6.2\bin\busybox.exe, sh. Install one and ensure it is available on the path.
during initialization of module BinaryProvider
while loading C:\Users\Chris\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Sundials\deps\build.jl, in expression starting on line 1
================================================================================
================================[ BUILD ERRORS ]================================
WARNING: Sundials had build errors.
- packages with build errors remain installed in C:\Users\Chris\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6
- build the package(s) and all dependencies with `Pkg.build("Sundials")`
- build a single package by running its `deps/build.jl` script
================================================================================
When you change the build script url, to e.g. use a new tag with new binaries, a user running Pkg.build("MyPackage")
will reuse whatever binary is already downloaded
When the download is corrupted, or from an older tag with non matching hashes, build fails, even though deleting and downloading the new binary would resolve the problem
Downloading from non existent urls works as if nothing is wrong, the empty file stays in the download folder and gets used, but fails with a hash not match error
Thanks for all the effort!
I think it's not uncommon for a make install
to create a directory structure like prefix/bin/libfoo/foo
or prefix/lib/libfoo/libfoo.so
(see attached screenshot for mariadb
) Not sure what the best way to handle this is. The BinaryBuilder is setup to just do a regular cmake && make && make install
, but then BinaryProvider has a hard time later because it's not located at usr/lib/libfoo.so
. I'm not sure if BinaryProvider could be more flexible here in recursing down a bit? Or should BinaryBuilder detect this kind of structure and complain? Or rearrange things itself?
This error popped up just after the build retry step of the BinaryBuilder wizard. Unfortunately the backtrace just shows the point where the task is being waited for, not where the actual error occured.
It looks like this might be this issue in Julia where reading a Char
from a Process
throws this error, which is what's happening here. The fix seems to be merged into 1.0 but still pending backporting for 0.7. Perhaps it's worth adding a workaround?
MethodError: no method matching position(::Pipe)
Closest candidates are:
position(::HTTP.Form) at /home/sfr/.julia/packages/HTTP/nUK4f/src/multipart.jl:23
position(::Base.SecretBuffer) at secretbuffer.jl:154
position(::Base.Filesystem.File) at filesystem.jl:225
...
Stacktrace:
[1] _wait(::Task) at ./task.jl:192
[2] fetch at ./task.jl:205 [inlined]
[3] wait(::OutputCollector) at /home/sfr/.julia/packages/BinaryProvider/ArZWB/src/OutputCollector.jl:155
[4] (::getfield(BinaryBuilder, Symbol("##78#80")){Bool,Base.TTY,BinaryBuilder.UserNSRunner,Cmd,String})() at /home/sfr/.julia/dev/BinaryBuilder/src/UserNSRunner.jl:126
[5] cd(::getfield(BinaryBuilder, Symbol("##78#80")){Bool,Base.TTY,BinaryBuilder.UserNSRunner,Cmd,String}, ::String) at ./file.jl:96
[6] #run#77(::Bool, ::Base.TTY, ::Function, ::BinaryBuilder.UserNSRunner, ::Cmd, ::String) at /home/sfr/.julia/dev/BinaryBuilder/src/UserNSRunner.jl:123
[7] (::getfield(Base, Symbol("#kw##run")))(::NamedTuple{(:verbose, :tee_stream),Tuple{Bool,Base.TTY}}, ::typeof(run), ::BinaryBuilder.UserNSRunner, ::Cmd, ::String) at ./none:0
[8] (::getfield(BinaryBuilder, Symbol("##217#218")){BinaryBuilder.WizardState,Linux,String})() at /home/sfr/.julia/dev/BinaryBuilder/src/wizard/interactive_build.jl:207
[9] cd(::getfield(BinaryBuilder, Symbol("##217#218")){BinaryBuilder.WizardState,Linux,String}, ::String) at ./file.jl:96
[10] step3_retry(::BinaryBuilder.WizardState) at /home/sfr/.julia/dev/BinaryBuilder/src/wizard/interactive_build.jl:192
[11] run_wizard(::BinaryBuilder.WizardState) at /home/sfr/.julia/dev/BinaryBuilder/src/Wizard.jl:60
[12] run_wizard() at /home/sfr/.julia/dev/BinaryBuilder/src/Wizard.jl:24
Running probe_plastform_engines!
takes about 0.6 seconds. Perhaps it is worth to cache the results as a build step? This would require rebuilding when a user reinstalls new engines but a 0.6-second startup seems a bit high for something that quite rarely changes?
I'm looking a bit into downloading packages as tarballs in Pkg3 and am currently using this package and this causes startup time to increase a bit too much. If this issue is not in line with how the package is thought to be used, I could, of course, roll my own build script.
Now, I'm not confident that BinaryProvider is at fault, but I would appreciate any suggestions on how to further diagnose and fix the issue.
For reference, I built Julia with:
$ gcc --version
gcc (Ubuntu 7.3.0-16ubuntu3) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Trying to debug by running code from here leads me to:
julia> libstdcxx_paths = filter(x -> occursin("libstdc++", x), dllist())
1-element Array{AbstractString,1}:
"/usr/lib/x86_64-linux-gnu/libstdc++.so.6"
julia> hdl = dlopen(first(libstdcxx_paths))
julia> dlsym_e(hdl, "GLIBCXX_3.4.25") # Null pointer
Ptr{Nothing} @0x0000000000000000
julia> dlsym_e(hdl, "GLIBCXX_3.4.13") # Null pointer
Ptr{Nothing} @0x0000000000000000
julia> max_version = v"3.4.0";
julia> for minor_version in 1:26
if dlsym_e(hdl, "GLIBCXX_3.4.$(minor_version)") != C_NULL
max_version = VersionNumber("3.4.$(minor_version)")
end
end
julia> max_version
v"3.4.0"
Trying to check the file from the command line...
$ nm -D /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX_3.4
0000000000000000 A GLIBCXX_3.4
0000000000000000 A GLIBCXX_3.4.1
0000000000000000 A GLIBCXX_3.4.10
0000000000000000 A GLIBCXX_3.4.11
0000000000000000 A GLIBCXX_3.4.12
0000000000000000 A GLIBCXX_3.4.13
0000000000000000 A GLIBCXX_3.4.14
0000000000000000 A GLIBCXX_3.4.15
0000000000000000 A GLIBCXX_3.4.16
0000000000000000 A GLIBCXX_3.4.17
0000000000000000 A GLIBCXX_3.4.18
0000000000000000 A GLIBCXX_3.4.19
0000000000000000 A GLIBCXX_3.4.2
0000000000000000 A GLIBCXX_3.4.20
0000000000000000 A GLIBCXX_3.4.21
0000000000000000 A GLIBCXX_3.4.22
0000000000000000 A GLIBCXX_3.4.23
0000000000000000 A GLIBCXX_3.4.24
0000000000000000 A GLIBCXX_3.4.25
0000000000000000 A GLIBCXX_3.4.3
0000000000000000 A GLIBCXX_3.4.4
0000000000000000 A GLIBCXX_3.4.5
0000000000000000 A GLIBCXX_3.4.6
0000000000000000 A GLIBCXX_3.4.7
0000000000000000 A GLIBCXX_3.4.8
0000000000000000 A GLIBCXX_3.4.9
So it contains the symbols up through 3.4.25. Although, each is also accompanied with what may be a null address on the left?
For example, if I don't grep, the last couple functions are:
000000000037bf60 V _ZTVSt9strstream
000000000037b490 V _ZTVSt9type_info
although the results are inconsistent with dlsym_e
:
julia> dlsym_e(hdl, "_ZTVSt9type_info")
Ptr{Nothing} @0x00007ff0ef183490
but perhaps that's because of the "V" (weak symbol object)?
Now, I also see the lines:
# Full list available here: https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html
if max_version < v"3.4.18"
error("libstdc++ ABI version '$max_version' too small; please build Julia with GCC >= 4.8.5")
elseif max_version < v"3.4.23"
# If we aren't up to 7.1.0, then we fall all the way back to 4.8.5
return :gcc4
elseif max_version < v"3.4.25"
return :gcc7
else
return :gcc8
end
which, because grep indicates it has v"3.4.25"
, which is not less than itself, it would be categorized as gcc8, when in fact it is gcc7, as shown above.
The link only states that v"3.4.24"
is for 7.2, not 7.3.
Although, maybe that means gcc 7.3 has gcc 8's libstdcxx_abi, in which case no problem. I know almost nothing about these things, and would appreciate clarification.
Where are the errors?
Should the addresses of the symbols listed by nm
not be 0? Is something wrong with my libstdc++.so.6
?
Or, is the null pointer check for them not a valid check to determine which version?
This was also a rather confusing error message to track down, because I had not built Julia with a gcc <= 4.8.5.
I reproduced this problem on another computer.
I know that Julia has been little-endian focused, and I agree with that. Now that people are starting to work with specialty data and compressed files generated with specialized hardware, allowing the user of this to state by keyword setting that the target (file or compressed collection of files) has been stored big-endian.
(I have no such need).
Sundials.jl works with the master branch of BinaryProvider. I was wondering if we could get a tag to really try this out. Thanks!
Busybox in single-executable mode, can be called as busybox wget
and busybox tar
. If busybox
is available, we should be able to use it like that, even if the individual symlinks are not available.
│ ERROR: LoadError: LoadError: LoadError: LoadError: UndefVarError: spawn not defined
juliav1.0.0rc1.0
With each Julia release, we should generate a BinaryProvider
-style tarball of shared libraries that typically ship with Julia. Things like libgfortran.so.4
, libstdc++.so.6
, etc... and enable automatic downloading of it to the julia private shlib location. These will then override the typical system libraries, and since they will be newer (and backwards-compatible) they should enable from-source builders of Julia to use the precompiled binaries that we ship through BinaryBuilder.
We will have to update this package of libraries every time the compilers we're using change something (a la libgfortran.so.3
-> libgfortran.so.4
in GCC 7) but hopefully we can keep those problems limited to a particular Julia major release.
I am trying to install EzXML, but the installation process failed with the error:
Error: Error building `EzXML`:
│ ERROR: LoadError: LibraryProduct(nothing, ["libxml2"], :libxml2, "Prefix(/home/ulg/gher/abarth/.julia/packages/EzXML/3U9Ow/deps/usr)") is not satisfied, cannot generate deps.jl!
The problem was that libxml2.so
require libz
, but the system libz
is too old.
It would be great if a user would see the error message of dlopen
, instead of failing silently.
https://github.com/JuliaPackaging/BinaryProvider.jl/blob/master/src/Products.jl#L172
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.