juliapackaging / binaryprovider.jl Goto Github PK

View Code? Open in Web Editor NEW

43.0 43.0 47.0 438 KB

A reliable binary provider for Julia

License: Other

Julia 99.44% Shell 0.56%

dependency-manager julia

binaryprovider.jl's People

Contributors

Stargazers

Watchers

binaryprovider.jl's Issues

Complain if the user tries to download using a mismatched GCC version

We should detect that e.g. libgfortran.5.dylib is loaded and freak out, refusing to download.

Hash Mismatch! Error.

I was trying to build Pkg.build("Sundials") when I got this error.

LoadError: Hash Mismatch! Expected sha256: f34ad4faa4d0c9841e15348a48c8ab5780af56da23af9d46274192c6bbcf1049 Calculated sha256: 4b5f9e8781909e722c51e134f68e83e6ca99ab612a5dab8ecc3e0f8057cfcdfd while loading /home/sohan/.julia/v0.6/Sundials/deps/build.jl, in expression starting on line 54

Can anyone tell what's causing this error ?

Note:- I'm behind a campus-proxy, but other julia packages get installed without a problem.
I'm on Ubuntu 16.04 LTS(Xenial amd64) using juila v0.6.0.

customize dlopen flags

In RingBuffers.jl I add the RTLD_GLOBAL flag to my dlopen call, so that C libraries from other packages (e.g. PortAudio.jl) can call the RingBuffer's C functions. BinaryProvider is definitely a slicker way to do what I was doing before, but it doesn't have a hook to override those flags.

Would a PR for that be accepted?

Doesn't find libz on FreeBSD for CodecZlib.jl

CodecZlib.jl requires libz: https://github.com/bicycle1885/CodecZlib.jl/blob/master/deps/build.jl

However, on FreeBSD it still downloads sources and breaks, even though /usr/lib/libz.so exists.

need import Compat.Libdl on 0.7

There seem to be several places that use Libdl.foo without first importing the Libdl module, which is required on 0.7. (For 0.6 compatibility, import Compat.Libdl). For example

here
here

The latter case is in generated code for deps.jl so it triggers deprecation warnings in packages (see e.g. JuliaIO/Blosc.jl#56).

Add proxy support to download function (PlatformEngines.jl::255+)

Suggest to include proxy support using default values on windows:
webclient_code = """
[System.Net.ServicePointManager]::SecurityProtocol =
[System.Net.SecurityProtocolType]::Tls12;
$webclient = (New-Object System.Net.Webclient);
$webclient.Headers.Add("user-agent", "$agent");
$webclient.Proxy.Credentials =[System.Net.CredentialCache]::DefaultNetworkCredentials;
$webclient.DownloadFile("$url", "$path")
"""

Could not extract the platform key

When building StringEncodings.jl, the log file shows a warning that says Could not extract the platform key

See details at JuliaStrings/StringEncodings.jl#31

curl error when installing a package

When installing CodecZlib (dependency decided by Pkg.update()), I got the following error:

curl: (48) An unknown option was passed in to libcurl

Here is the full log when trying to build only this package:

INFO: Building CodecZlib
Info: Downloading https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.0/Zlib.x86_64-linux-gnu.tar.gz to /raid/home/thibaut/.julia/v0.6/CodecZlib/deps/usr/downloads/Zlib.x86_64-linux-gnu.tar.gz...
[16:28:38] curl: (48) An unknown option was passed in to libcurl
==========================================================================[ ERROR: CodecZlib ]==========================================================================

LoadError: Could not download https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.0/Zlib.x86_64-linux-gnu.tar.gz to /raid/home/thibaut/.julia/v0.6/CodecZlib/deps/usr/downloads/Zlib.x86_64-linux-gnu.tar.gz
while loading /raid/home/thibaut/.julia/v0.6/CodecZlib/deps/build.jl, in expression starting on line 92

========================================================================================================================================================================

============================================================================[ BUILD ERRORS ]============================================================================

WARNING: CodecZlib had build errors.

 - packages with build errors remain installed in /raid/home/thibaut/.julia/v0.6
 - build the package(s) and all dependencies with `Pkg.build("CodecZlib")`
 - build a single package by running its `deps/build.jl` script

========================================================================================================================================================================

What's strange is that I have the latest version of curl installed:

curl 7.59.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.0.2n zlib/1.2.11
Release-Date: 2018-03-14
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy

I am using Julia 0.6.2:

   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.2 (2017-12-13 18:08 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-pc-linux-gnu

julia> versioninfo()
Julia Version 0.6.2
Commit d386e40c17 (2017-12-13 18:08 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, ivybridge)

Unable to install Ipopt using BinaryProvider.jl

Hi all,

I'm unable to install Ipopt and need some assistance. When I run build Ipopt (I get the same error with add Ipopt and add Ipopt#master), I get the following error on a fresh julia install:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.0.1 (2018-09-29)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(v1.0) pkg> add Ipopt
   Cloning default registries into /Users/$USER/.julia/registries
   Cloning registry General from "https://github.com/JuliaRegistries/General.git"
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
 Installed BinDeps ────────── v0.8.10
 Installed Compat ─────────── v1.3.0
 Installed Ipopt ──────────── v0.4.2
 Installed MathProgBase ───── v0.7.7
 Installed BinaryProvider ─── v0.5.2
 Installed MathOptInterface ─ v0.6.1
 Installed URIParser ──────── v0.4.0
  Updating `~/.julia/environments/v1.0/Project.toml`
  [b6b21f68] + Ipopt v0.4.2
  Updating `~/.julia/environments/v1.0/Manifest.toml`
  [9e28174c] + BinDeps v0.8.10
  [b99e7846] + BinaryProvider v0.5.2
  [34da2185] + Compat v1.3.0
  [b6b21f68] + Ipopt v0.4.2
  [b8f27783] + MathOptInterface v0.6.1
  [fdba3010] + MathProgBase v0.7.7
  [30578b45] + URIParser v0.4.0
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  Building Ipopt → `~/.julia/packages/Ipopt/5a2PI/deps/build.log`
┌ Error: Error building `Ipopt`:
│ ┌ Warning: platform_key() is deprecated, use platform_key_abi() from now on
│ │   caller = ip:0x0
│ └ @ Core :-1
│ ┌ Warning: Could not extract the platform key of https://github.com/staticfloat/IpoptBuilder/releases/download/v3.12.8-9/Ipopt.x86_64-apple-darwin14.tar.gz; continuing...
│ └ @ BinaryProvider ~/.julia/packages/BinaryProvider/1nGWd/src/Prefix.jl:185
│ [ Info: Downloading https://github.com/staticfloat/IpoptBuilder/releases/download/v3.12.8-9/Ipopt.x86_64-apple-darwin14.tar.gz to /Users/$USER/.julia/packages/Ipopt/5a2PI/deps/usr/downloads/Ipopt.x86_64-apple-darwin14.tar.gz...
│ ERROR: LoadError: LibraryProduct(nothing, ["libipopt"], :libipopt, "Prefix(/Users/$USER/.julia/packages/Ipopt/5a2PI/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{Product,1}) at /Users/$USER/.julia/packages/BinaryProvider/1nGWd/src/Products.jl:408
│  [3] write_deps_file(::String, ::Array{Product,1}) at /Users/$USER/.julia/packages/BinaryProvider/1nGWd/src/Products.jl:395
│  [4] top-level scope at /Users/$USER/.julia/packages/Ipopt/5a2PI/deps/build.jl:51
│  [5] include at ./boot.jl:317 [inlined]
│  [6] include_relative(::Module, ::String) at ./loading.jl:1041
│  [7] include(::Module, ::String) at ./sysimg.jl:29
│  [8] include(::String) at ./client.jl:388
│  [9] top-level scope at none:0
│ in expression starting at /Users/$USER/.julia/packages/Ipopt/5a2PI/deps/build.jl:13
[12:32:07] ######################################################################## 100.0%
└ @ Pkg.Operations /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1069

(v1.0) pkg>

Question 1: What is /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1069? I don't have a osx user on my machine. Where is this hardcoded message coming from? The Julia base image?

This tar.gz file that is being downloaded in this process is located here.

If I cd into the Ipopt folder and run otool on the libraries, I see the following:

otool -L ~/.julia/packages/Ipopt/5a2PI/deps/usr/lib/libipopt.dylib
/Users/$USER/.julia/packages/Ipopt/5a2PI/deps/usr/lib/libipopt.dylib:
        /workspace/destdir/lib/libipopt.1.dylib (compatibility version 12.0.0, current version 12.8.0)
        @loader_path/libcoinmumps.1.dylib (compatibility version 8.0.0, current version 8.0.0)
        @rpath/libgfortran.4.dylib (compatibility version 5.0.0, current version 5.0.0)
        @rpath/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
        /opt/x86_64-apple-darwin14/x86_64-apple-darwin14/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.24.0)

Question 2: Why is there a /opt/x86_64-apple-darwin14/x86_64-apple-darwin14/lib/libstdc++.6.dylib? Shouldn't this also use @rpath? Or at least point to /usr/lib/libc++.1.dylib instead?

This is what happens when I run julia and try to load the library.

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.0.1 (2018-09-29)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> import Libdl

julia> Libdl.dlopen("./libipopt.dylib")
ERROR: could not load library "./libipopt.dylib"
dlopen(./libipopt.dylib, 1): Library not loaded: /opt/x86_64-apple-darwin14/x86_64-apple-darwin14/lib/libstdc++.6.dylib
  Referenced from: /Users/$USER/.julia/packages/Ipopt/5a2PI/deps/usr/lib/libipopt.1.10.8.dylib
  Reason: image not found
Stacktrace:
 [1] dlopen(::String, ::UInt32) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Libdl/src/Libdl.jl:97 (repeats 2 times)
 [2] top-level scope at none:0

julia>

Any thoughts on how to fix this issue? It appears to work on other colleagues' Mac machines.

Use `using Compat.Sys: iswindows()`

Better solution than living with deprecation warnings

add url2hash for developper convenience

When putting together install/build scripts, it's a bit inconvenient and not entirely clear how to get the correct hash. After digging around in BinaryProvider I came up with the following function to get a hash:

"""
     url2hash(url::String)

usage:
     hash = url2hash("https://downloads.xiph.org/releases/vorbis/libvorbis-1.3.6.tar.xz")
"""
function url2hash(url::String)
    path = download(url)
    open(io-> bytes2hex(BinaryProvider.sha256(io)), path)
end

Maybe we can add this somewhere to BinaryProvider?

0.7 error

┌ Warning: `warn()` is deprecated, use `@warn` instead.
│   caller = #warn#872(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::String, ::Vararg{String,N} where N) at deprecated.jl:1010
└ @ Base deprecated.jl:1010
WARNING: Could not spawn `curl -C - '-#' -f -o /home/travis/.julia/v0.7/MbedTLS/deps/usr/downloads/MbedTLS.x86_64-linux-gnu.tar.gz -L https://github.com/quinnj/MbedTLSBuilder/releases/download/v0.5/MbedTLS.x86_64-linux-gnu.tar.gz`
┌ Error: ------------------------------------------------------------
│ # Build failed for MbedTLS
│   exception =
│    LoadError: MethodError: no method matching spawn(::Cmd, ::Tuple{Base.DevNullStream,Pipe,Pipe})
│    Closest candidates are:
│      spawn(::Base.AbstractCmd) at deprecated.jl:55
│    Stacktrace:
│     [1] #OutputCollector#5(::Bool, ::Bool, ::Base.TTY, ::Type, ::Cmd) at /home/travis/.julia/v0.7/BinaryProvider/src/OutputCollector.jl:115
│     [2] (::getfield(Core, Symbol("#kw#Type")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::Type{BinaryProvider.OutputCollector}, ::Cmd) at ./<missing>:0
│     [3] #download#89(::Bool, ::Function, ::String, ::String) at /home/travis/.julia/v0.7/BinaryProvider/src/PlatformEngines.jl:421

from this log: https://travis-ci.org/JuliaWeb/MbedTLS.jl/jobs/348374182

symbol renaming to avoid library conflicts

It might be nice to provide an easy way to rename all the symbols in a library, to avoid conflicts with other external libraries like we encountered in JuliaLang/julia#4923.

For example, if you included GNU objcopy in your cross-compile setup, then something like --prefix-symbols could be used for this without modifying any source code.

look for .dll files in the lib directory

Currently, BinaryProvider assumes that .dll files are installed into the bin directory. If they are installed into lib, I get an error like this:

LoadError: BinaryProvider.LibraryProduct("C:\\Users\\appveyor\\.julia\\v0.6\\Blosc\\deps\\usr\\bin", String["libblosc"], :libblosc, "Prefix(C:\\Users\\appveyor\\.julia\\v0.6\\Blosc\\deps\\usr)") is not satisfied, cannot generate deps.jl!
while loading C:\Users\appveyor\.julia\v0.6\Blosc\deps\build.jl, in expression starting on line 40

As a workaround, I notice that e.g. MbedTLSBuilder's build_tarballs.jl script has a hack to copy .dll files to the bin directory:

if [ $target == "x86_64-w64-mingw32" ]; then
    cp $prefix/lib/*.dll $prefix/bin/.
elif [ $target == "i686-w64-mingw32" ]; then
    cp $prefix/lib/*.dll $prefix/bin/.

Couldn't BinaryProvider look in both the lib and bin directories for .dll files on Windows?

(Or alternatively BinaryBuilder's build_tarballs function could move .dll files into the bin directory.)

It's pretty common for Unixy makefiles to install all libraries into lib, even when cross-compiling, and it seems unfortunate to require every build_tarballs.jl to include some boilerplate code to move the .dll files to a different directory.

Changing compiler ABI

I've got a working BinaryBuilder release for my package here: https://github.com/BenJWard/BSGBuilder/releases/tag/v0.6.0

When I use the build.jl file in a package on my local Mojave macbook pro, the following happens:

(v1.0) pkg> activate .
(BSG) pkg> add BinaryProvider
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
  Updating `~/Desktop/BSG/Project.toml`
  [b99e7846] + BinaryProvider v0.5.2
  Updating `~/Desktop/BSG/Manifest.toml`
 [no changes]

(BSG) pkg> build
  Building CxxWrap → `~/.julia/packages/CxxWrap/KcmSi/deps/build.log`
  Building BSG ────→ `~/Desktop/BSG/deps/build.log`
 Resolving package versions...
┌ Error: Error building `BSG`: 
│ ERROR: LoadError: ArgumentError: Unable to find matching download for MacOS(:x86_64, compiler_abi=CompilerABI(:gcc8, :cxx03))
│ Stacktrace:
│  [1] choose_download(::Dict{Platform,Tuple{String,String}}, ::MacOS) at /Users/bward/.julia/packages/BinaryProvider/1nGWd/src/PlatformNames.jl:733
│  [2] top-level scope at none:0
│  [3] include at ./boot.jl:317 [inlined]
│  [4] include_relative(::Module, ::String) at ./loading.jl:1041
│  [5] include(::Module, ::String) at ./sysimg.jl:29
│  [6] include(::String) at ./client.jl:388
│  [7] top-level scope at none:0
│ in expression starting at /Users/bward/Desktop/BSG/deps/build.jl:24
└ @ Pkg.Operations ~/github/julia/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1069

My binary releases only use cxx11 and either gcc8 or gcc7 as their compiler_abi options, because bsg really needs cxx11 as a minimum. Is there a way to change the cxx standard options when downloading a build?

The identified package name in check_deps is wrong

To reproduce, Pkg.add("SpecialFunctions") then take a look at the generated deps.jl file. Mine looks like

## This file autogenerated by BinaryProvider.write_deps_file().
## Do not edit.
##
## Include this file within your main top-level source, and call
## `check_deps()` from within your module's `__init__()` method

if isdefined((@static VERSION < v"0.7.0-DEV.484" ? current_module() : @__MODULE__), :Compat)
    import Compat.Libdl
elseif VERSION >= v"0.7.0-DEV.3382"
    import Libdl
end
const openspecfun = joinpath(dirname(@__FILE__), "usr/lib/libopenspecfun.1.3.dylib")
function check_deps()
    global openspecfun
    if !isfile(openspecfun)
        error("$(openspecfun) does not exist, Please re-run Pkg.build(\"KvXoO\"), and restart Julia.")
    end

    if Libdl.dlopen_e(openspecfun) in (C_NULL, nothing)
        error("$(openspecfun) cannot be opened, Please re-run Pkg.build(\"KvXoO\"), and restart Julia.")
    end

end

Note in particular

Please re-run Pkg.build(\"KvXoO\")

This is due to this line:

BinaryProvider.jl/src/Products.jl

Line 398 in d0e7869

package_name = basename(dirname(dirname(depsjl_path)))

which is not quite right. In my case the path is /Users/alex/.julia/packages/SpecialFunctions/KvXoO/deps/deps.jl.

Deadlock on Windows on spawning multiple tasks of download and unpack

JuliaLang/Pkg.jl#25 attempts to implement asynchronous downloading and unpacking of tarballs for getting the packages.

There was some problems on Windows that resulted in a deadlock

I have made a branch here (with some debug output) https://github.com/KristofferC/Pkg3.jl/tree/kc/winrepro that for me reproduces the problem when running

julia> import Pkg3

<]> # enter pkg repl

pkg> install GLVisualize
.....
......
****************
Packages left to handle:
IdentityRanges scheduled: true
INFO: Installed CoordinateTransformations ─ v0.4.1
[DEADLOCK]

What the debug output is saying is that we have scheduled the package "IdentityRanges" to be downloaded and unpacked here: https://github.com/KristofferC/Pkg3.jl/blob/kc/winrepro/src/Operations.jl#L379 but the put! statement on the next line is never reached. In the function install (https://github.com/KristofferC/Pkg3.jl/blob/kc/winrepro/src/Operations.jl#L248) the acrhive is downloaded and uncompressed. Further debugging suggested that we are stuck waiting for the OutputCollector task to finish and it never does, which is why the blocking take! call at https://github.com/KristofferC/Pkg3.jl/blob/kc/winrepro/src/Operations.jl#L395 never finishes.

Moving to just downloading and unpacking using the generated commands from BinaryProvider worked around the issue, so perhaps there is some problem with the OutputCollector.

I am sorry to not be able to provide a more minimum example. The issue is quite finicky since depending on where I put print statements the pkg command might succeed or not.

FWIW, It seems that it is usually the first task that gets scheduled that gets stuck, if that can help debugging...

Interrupting a download (^C)

Perhaps this could be considered as using internals of the package, but anyway:

When Interrupting a download with e.g. ^C I get:

^CERROR (unhandled task failure): InterruptException:
Stacktrace:
 [1] wait() at /Applications/Julia-0.6.app/Contents/Resources/julia/lib/julia/sys.dylib:? (repeats 3 times)
 [2] macro expansion at .../BinaryProvider/src/OutputCollector.jl:59 [inlined]
 [3] (::BinaryProvider.##2#4{Condition,Task})() at ./task.jl:335

and the call to download then throws a normal ErrorException that the download failed. In a situation where you try to download a file from different urls, you probably want to respond differently to a failure to download a file, and a ^C from the user.

Would it be possible to "handle the task failure" and propagate the InterruptException out to the caller of download?

Use `access()` instead of stat's mode

via @ararslan:

"""
    isexecutable(program)

Determine whether the given program name or path is executable using the current user's
permissions. This is roughly equivalent to querying `which program` at the command line
and checking that a result is found, but no shelling out occurs.
"""
function isexecutable(k::Symbol, prog::String)
    X_OK = 1 << 0 # Taken from unistd.h
    # If prog has a slash, we know the user wants to determine whether the given
    # file exists and is executable
    if '/' in prog || (iswindows(k) && '\\' in prog) # Windows can use / too
        isfile(prog) || return false
        return ccall(:access, Cint, (Ptr{UInt8}, Cint), prog, X_OK) == 0
    end
    path = get(ENV, "PATH", "")
    # Something is definitely wrong if the user's path is empty...
    @assert !isempty(path)
    sep = iswindows(k) ? ';' : ':'
    for dir in split(path, sep), file in readdir(dir)
        if file == prog || (iswindows(k) && file == prog * ".exe")
            p = joinpath(dir, file)
            @assert isfile(p)
            return ccall(:access, Cint, (Ptr{UInt8}, Cint), p, X_OK) == 0
        end
    end
    false
end
isexecutable(prog::String) = isexecutable(Sys.KERNEL, prog)

locate() is greedy when matching library name

When called from write_deps_file(), locate() will happily match any library that begins with the same string as the library we are looking for:

BinaryProvider.jl/src/Products.jl

Line 158 in cb403e6

if startswith(basename(f), libname)

In my case I was trying to use the library libgsl, which existed in the same folder as the library libgslcblas:

Info: Found a valid dl path libgslcblas.so while looking for libgsl
Info: /home/ludvigak/.julia/dev/GSL/deps/usr/lib/libgslcblas.so matches our search criteria of libgsl

I could fix this by changing
LibraryProduct(prefix, ["libgsl"], :libgsl)
to
LibraryProduct(prefix, ["libgsl."], :libgsl)
in the build script, but that was not obvious from the start.

This greedy matching could be considered a feature or a bug, depending on how you want things to work.

Could deps.jl use relative paths instead of absolute?

Ran into an issue today trying to deploy an app where my julia image was built in our build pipeline (jenkins), including building/precompiling packages like MbedTLS.jl, but then when pushed to the actual deploy server fails because the deps.jl file for MbedTLS.jl was hard-coded w/ the absolute path from the build node. Could we just use relative paths in the generated deps.jl file so we don't run into these issues? Currently I'm having to "rebuild" on each node.

Allow for download resuming

When downloading large files, (especially over intermittent connections) it should be possible to use curl's "resume" functionality to continue interrupted downloads.

BinaryProvider can install a library on Windows with Julia 0.7 but not with 0.6

I'm not sure this is a bug in this package, I opened a thread on discourse yesterday but got no reply. What I can't understand is why it works with Julia 0.7, but not with 0.6. Thus I suspect it might be a bug (or an error on my side that I can't see). In addition, I don't have a Windows box to play with, so it's a bit hard for me to track down the issue.

Below my issue report.

I'm trying to switch a package to BinaryBuilder.jl. It works fine on most systems (at least, those that I can test with Travis), but I'm facing some difficulties with the Windows build: the library is successfully installed in Julia 0.7, but fails with Julia 0.6 with the following error:

LoadError: BinaryProvider.LibraryProduct(nothing, String["libwcs"], :libwcs, "Prefix(C:\\Users\\appveyor\\.julia\\v0.6\\WCS\\deps\\usr)") is not satisfied, cannot generate deps.jl!
while loading C:\Users\appveyor\.julia\v0.6\WCS\deps\build.jl, in expression starting on line 46

Here are some information:

the script used to build the library: https://github.com/JuliaAstro/WCSLIBBuilder/blob/43b1ad73e51c45fb9975c0147c15ecef1011c815/build_tarballs.jl
the tag where the built packages are stored: https://github.com/JuliaAstro/WCSLIBBuilder/releases/tag/v5.13a
the build.jl script in the Julia wrapper: https://github.com/giordano/WCS.jl/blob/b570c9004682ff6fd71bb1f0728416091cf3f7ca/deps/build.jl
AppVeyor: https://ci.appveyor.com/project/giordano/wcs-jl/build/1.0.6
- Failure to install the library with Julia 0.6: https://ci.appveyor.com/project/giordano/wcs-jl/build/1.0.6/job/em7v7em5ug5rmnxq
- Success with Julia 0.7: https://ci.appveyor.com/project/giordano/wcs-jl/build/1.0.6/job/4mavkajcido8kpbg

Has anyone a clue about why Julia 0.6 fails to install the library on Windows? Am I doing something wrong, maybe in the build script?

Fails to find libLLVM on anaconda

I'm not sure if this is a problem with the conda-forge julia package or this package, but using a package that requires BinaryProvider.jl with Anaconda fails to find the libLLVM symbols.

On a Linux system with docker, this can be reproduced by running:

docker run continuumio/anaconda /bin/bash -c "/opt/conda/bin/conda install -c conda-forge -yq julia && julia -e 'using Pkg; Pkg.add(\"ZMQ\")'"

This results in the following error:

  Building ZMQ → `~/.julia/packages/ZMQ/ABGOx/deps/build.log`
┌ Error: Error building `ZMQ`: 
│ ERROR: LoadError: LoadError: Unable to open libLLVM!
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] (::getfield(BinaryProvider, Symbol("#open_libllvm#114")))() at /root/.julia/packages/BinaryProvider/cVlaj/src/PlatformNames.jl:649
│  [3] detect_cxx11_string_abi() at /root/.julia/packages/BinaryProvider/cVlaj/src/PlatformNames.jl:652
│  [4] detect_compiler_abi() at /root/.julia/packages/BinaryProvider/cVlaj/src/PlatformNames.jl:665
│  [5] top-level scope at none:0
│  [6] include at ./boot.jl:317 [inlined]
│  [7] include_relative(::Module, ::String) at ./loading.jl:1038
│  [8] include at ./sysimg.jl:29 [inlined]
│  [9] include(::String) at /root/.julia/packages/BinaryProvider/cVlaj/src/BinaryProvider.jl:1
│  [10] top-level scope at none:0
│  [11] include at ./boot.jl:317 [inlined]
│  [12] include_relative(::Module, ::String) at ./loading.jl:1038
│  [13] include(::Module, ::String) at ./sysimg.jl:29
│  [14] top-level scope at none:2
│  [15] eval at ./boot.jl:319 [inlined]
│  [16] eval(::Expr) at ./client.jl:389
│  [17] top-level scope at ./none:3
│ in expression starting at /root/.julia/packages/BinaryProvider/cVlaj/src/PlatformNames.jl:682
│ in expression starting at /root/.julia/packages/BinaryProvider/cVlaj/src/BinaryProvider.jl:12
│ ERROR: LoadError: Failed to precompile BinaryProvider [b99e7846-7c00-51b0-8f62-c81ae34c0232] to /root/.julia/compiled/v1.0/BinaryProvider/ek6VZ.ji.
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] macro expansion at ./logging.jl:313 [inlined]
│  [3] compilecache(::Base.PkgId, ::String) at ./loading.jl:1184
│  [4] _require(::Base.PkgId) at ./logging.jl:311
│  [5] require(::Base.PkgId) at ./loading.jl:852
│  [6] macro expansion at ./logging.jl:311 [inlined]
│  [7] require(::Module, ::Symbol) at ./loading.jl:834
│  [8] include at ./boot.jl:317 [inlined]
│  [9] include_relative(::Module, ::String) at ./loading.jl:1038
│  [10] include(::Module, ::String) at ./sysimg.jl:29
│  [11] include(::String) at ./client.jl:388
│  [12] top-level scope at none:0
│ in expression starting at /root/.julia/packages/ZMQ/ABGOx/deps/build.jl:1
└ @ Pkg.Operations /home/conda/feedstock_root/build_artifacts/julia_1534540929950/work/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1068

This was not the case for Julia 0.6.1, although that used BinaryProvider v0.3.3

Add hash files

When verifying downloads, save hashes to a $(file).sha256 file, then just check mtime timestamps to ensure that a large file hasn't changed since the last time we verified, so that we aren't constantly verifying multi-GB rootfs images in BinaryBuilder.

Run femtocleaner

Please

don't install into package directory

Currently, BinaryProvider downloads into MyPackage/deps/downloads and installs into MyPackage/deps/usr. With Julia 1.0, every version of MyPackage installs into its own directory, which means that every update installs a new copy of the binary dependencies even if they didn't change.

I don't think this is acceptable in the long run. Proposal:

BinaryProvider should install binary dependencies into a global SHA-addressed directory, e.g. ~/.julia/BinaryProvider/{MyPackage-UUID}/sha, so that they are shared between versions.
Some kind of hook should be added to Pkg.gc() to allow binary dependencies that are no longer needed to be deleted.

platform_key for ARM

Came across this in GEOSBuilder, building with TARGET=arm-linux-gnueabihf.
https://travis-ci.org/JuliaGeo/GEOSBuilder/jobs/360575777#L507-L516

julia> triplet(BinaryProvider.Linux(:armv7l))
"arm-linux-gnueabihf"

julia> platform_key("arm-linux-gnueabihf")
WARNING: Platform `arm-linux-gnueabihf` is not an officially supported platform
BinaryProvider.UnknownPlatform()

julia> platform_key("armv7l-linux-gnueabihf")
BinaryProvider.Linux(:armv7l, :glibc, :eabihf)

This can be reversed by chaning the dict in platform_key to :armv7l => "arm", but I'm not sure if this is the way to go, or that we need do deal with sub-arch explicitly.

Not possible to install Sundials on Windows due to creation of symbolic links (admin rights needed)

Downstream issue: SciML/Sundials.jl#172 . CC @MartinOtter.

under Windows 7 and 10 creating symbolic links requires administrator rights (that I do not have). Would it be possible to change the setup-procedures so that no symbolic links are set?

Too many log messages when Prefix points to a system library path

When I edit build.jl so that it finds libxml2 from "/usr/lib/" and run it to generate deps.jl, it emits too many log messages that are not so informative (I know the directory includes such libraries in it):

prefix = Prefix(!isempty(ARGS) ? ARGS[1] : joinpath(@__DIR__, "usr"))

↓

prefix = Prefix("/usr")

~/w/EzXML (master|✚1…) $ julia-dev deps/build.jl
┌ Warning: `ismatch(r::Regex, s::AbstractString)` is deprecated, use `contains(s, r)` instead.
│   caller = valid_dl_path(::String, ::MacOS) at PlatformNames.jl:204
└ @ BinaryProvider PlatformNames.jl:204
[ Info: Found a valid dl path PN548_API.dylib while looking for libxml2
[ Info: Found a valid dl path PN548_HAL_OSX.dylib while looking for libxml2
[ Info: Found a valid dl path PN548_OSX.dylib while looking for libxml2
[ Info: Found a valid dl path libATCommandStudioDynamic.dylib while looking for libxml2
[ Info: Found a valid dl path libAccountPolicyTranslation.dylib while looking for libxml2
[ Info: Found a valid dl path libBSDPClient.A.dylib while looking for libxml2
[ Info: Found a valid dl path libBSDPClient.dylib while looking for libxml2
[ Info: Found a valid dl path libCRFSuite.dylib while looking for libxml2
[ Info: Found a valid dl path libCRFSuite0.12.dylib while looking for libxml2
[ Info: Found a valid dl path libChineseTokenizer.dylib while looking for libxml2
[ Info: Found a valid dl path libCoreStorage.dylib while looking for libxml2
᠁
more than 200 lines of messages here
᠁
[ Info: Found a valid dl path libxml2.2.dylib while looking for libxml2
[ Info: /usr/lib/libxml2.2.dylib matches our search criteria of libxml2
WARNING: importing deprecated binding Base.Libdl into Sys.
WARNING: Base.Libdl is deprecated, run `using Libdl` instead
  likely near /Users/kenta/workspace/EzXML/deps/build.jl:26

I think these messages are astonishing but not so useful, so it should be less noisy.

Run femtocleaner

Windows install error

Appveyor log: https://ci.appveyor.com/project/quinnj/mbedtls-jl/build/1.0.110/job/pspktnckw9kyftfr

MacOS doesn't delete old binaries when updating

Multiple reports which seem to be this same issue that users from the old Sundials on MacOS get both binaries when it updates, and that confuses BinaryProvider.jl

Symbolic links on Windows

This is speculative as I am not an expert here.

On my Win10 unpack fails because creation of symbolic links throws an error that is not caught.
Maybe on Win10 ENV["BINARYPROVIDER_COPYDEREF"] should be set to "true" by default on this platform as this solves the problem.

I am not sure if this is my local problem or a general problem for Win10.

BinaryProvider doesn't detect that FreeBSD already has what's needed for CodecXz

(Related to issues #32 and #57)
My suggestion is to have the build.jl script, here we're using CodecXz as an example, somehow check whether the library it's looking for already exists on the system, and if so, proceed directly to creating the deps.jl file. In this specific case, FreeBSD has /usr/lib/liblzma.so.

I made CodecXz work by manually creating a deps.jl file with the right path to the library.

BinaryProvider.download not checking for Windows Management Framework

On a Win 7 machine:

julia> using BinaryProvider

┌ Warning: On Windows, creating file symlinks requires Administrator privileges

└ @ Base.Filesystem file.jl:789

 

julia> BinaryProvider.download("https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.2/Zlib.v1.2.11.x86_64-

w64-mingw32.tar.gz","zlib.tar.gz")

Exception setting "SecurityProtocol": "Cannot convert null to type "System.Net.SecurityProtocolType" due to invalid enu

meration values. Specify one of the following enumeration values and try again. The possible enumeration values are "Ss

l3, Tls"."

At line:1 char:35

+ [System.Net.ServicePointManager]:: <<<< SecurityProtocol =

    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException

    + FullyQualifiedErrorId : PropertyAssignmentException

 

Exception calling "DownloadFile" with "2" argument(s): "The underlying connection was closed: An unexpected error occur

red on a send."

At line:5 char:24

+ $webclient.DownloadFile <<<< ("https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.2/Zlib.v1.2.11.x86_6

4-w64-mingw32.tar.gz", "zlib.tar.gz")

    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException

    + FullyQualifiedErrorId : DotNetMethodException

 

ERROR: Could not download https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.2/Zlib.v1.2.11.x86_64-w64-mi

ngw32.tar.gz to zlib.tar.gz:

ErrorException("")

Stacktrace:

[1] error(::String) at .\error.jl:33

[2] #download#93(::Bool, ::Function, ::String, ::String) at .\logging.jl:313

[3] download(::String, ::String) at C:\Users\sbyrne\.julia\packages\BinaryProvider\cVlaj\src\PlatformEngines.jl:485

[4] top-level scope at none:0

This gives confusing errors, e.g. JuliaIO/CodecZlib.jl#24 (comment)

Base.download checks for this:

julia> download("https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.2/Zlib.v1.2.11.x86_64-w64-mingw32.tar

.gz")

Cannot start Windows PowerShell version 3 because it is not correctly installed.

ERROR: Downloading files requires Windows Management Framework 3.0 or later.

Stacktrace:

[1] error(::String) at .\error.jl:33

[2] download(::String, ::String) at .\download.jl:20

[3] download(::String) at .\download.jl:59

[4] top-level scope at none:

re-download on Pkg.build if hash changes

Currently, the build.jl logic does nothing if the satisfied returns true for all the build products, i.e. if the libraries are all found in deps/usr. This makes it impossible to update the binaries without either manually doing Pkg.rm(Pkg.dir("deps", "usr"), recursive=true) or renaming the build products.

This seems like it makes it very hard to push fixes in the binaries to users.

Proposal: Pkg.build should always re-download if the hash changes. In particular the build.jl logic could be changed to something like:

unsatisfied = any(!satisfied(p; verbose=verbose) for p in products)
if haskey(download_info, platform_key())
    url, tarball_hash = download_info[platform_key()]
    if unsatisfied || !isinstalled(tarball_hash)
        install(url, tarball_hash; prefix=prefix, force=true, verbose=verbose)
    end
elseif unsatisfied
    error("Your platform $(triplet(platform_key())) is not supported by this package!")
end

where BinaryProvider.isinstalled(tarball_hash) checks some file that saves the hash.

Failing test on windows in the grep use

Error in tests on 0.7 on win64 (binary download) in runtest.jl,L101.

100    # Next, test reading the output of a pipeline()
101    grepline = pipeline(sh(`-c 'printf "Hello\nWorld\nJulia"'`), `grep ul`)
102    oc = OutputCollector(grepline)

The problem is, the grep ul command is not "wrapped" with busybox but windows tries to find some native "grep.exe" that doesn't generally exist (of course you might be lucky and windows can find some somewhere on path, e.g. from the git install - this is why the test could pass on travis or locally).

Full stacktrace here:

[ Info: Probing C:\Julia-0.7.0\bin\busybox.exe as a possibility...
[ Info:   Probe successful for C:\Julia-0.7.0\bin\busybox.exe
[ Info: Found sh engine C:\Julia-0.7.0\bin\busybox.exe
1
2
┌ Warning: Could not spawn pipeline(`'C:\Julia-0.7.0\bin\busybox.exe' sh -c -c 'printf "Hello\nWorld\nJulia"'`, stdout=`grep ul`)
└ @ BinaryProvider C:\Users\phlavenk\.julia\packages\BinaryProvider\ZbFxn\src\OutputCollector.jl:120
OutputCollector: Error During Test at C:\Users\phlavenk\.julia\packages\BinaryProvider\ZbFxn\test\runtests.jl:33
  Got exception Base.IOError("could not spawn `grep ul`: no such file or directory (ENOENT)", -4058) outside of a @test
  IOError: could not spawn `grep ul`: no such file or directory (ENOENT)
  Stacktrace:
   [1] _jl_spawn(::String, ::Array{String,1}, ::Cmd, ::Tuple{Base.Libc.WindowsRawSocket,Base.PipeEndpoint,Base.PipeEndpoint}) at .\process.jl:370
   [2] (::getfield(Base, Symbol("##499#500")){Cmd})(::Tuple{Base.Libc.WindowsRawSocket,Base.PipeEndpoint,Base.PipeEndpoint}) at .\process.jl:512
   [3] setup_stdio(::getfield(Base, Symbol("##499#500")){Cmd}, ::Tuple{Base.Libc.WindowsRawSocket,Pipe,Pipe}) at .\process.jl:493
   [4] #_spawn#498(::Base.ProcessChain, ::Function, ::Cmd, ::Tuple{Base.Libc.WindowsRawSocket,Pipe,Pipe}) at .\process.jl:511
   [5] (::getfield(Base, Symbol("#kw##_spawn")))(::NamedTuple{(:chain,),Tuple{Base.ProcessChain}}, ::typeof(Base._spawn), ::Cmd, ::Tuple{Base.Libc.WindowsRawSocket,Pipe,Pipe}) at .\none:0
   [6] #_spawn#496(::Nothing, ::Function, ::Base.OrCmds, ::Tuple{Base.DevNullStream,Pipe,Pipe}) at .\process.jl:415

   [7] (::getfield(Base, Symbol("#kw##_spawn")))(::NamedTuple{(:chain,),Tuple{Nothing}}, ::typeof(Base._spawn), ::Base.OrCmds, ::Tuple{Base.DevNullStream,Pipe,Pipe}) at .\none:0
   [8] #_spawn#495(::Nothing, ::Function, ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Pipe,Pipe}) at .\process.jl:401
   [9] (::getfield(Base, Symbol("#kw##_spawn")))(::NamedTuple{(:chain,),Tuple{Nothing}}, ::typeof(Base._spawn), ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Pipe,Pipe}) at .\none:0
   [10] #_spawn#495(::Nothing, ::Function, ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Base.DevNullStream,Pipe}) at .\process.jl:401
   [11] (::getfield(Base, Symbol("#kw##_spawn")))(::NamedTuple{(:chain,),Tuple{Nothing}}, ::typeof(Base._spawn), ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Base.DevNullStream,Pipe}) at .\none:0
   [12] #_spawn#495(::Nothing, ::Function, ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Base.DevNullStream,Base.DevNullStream}) at .\process.jl:401
   [13] _spawn at .\process.jl:401 [inlined]
   [14] #run#509(::Bool, ::Function, ::Base.CmdRedirect) at .\process.jl:672
   [15] #run at .\none:0 [inlined]
   [16] #OutputCollector#5(::Bool, ::Bool, ::Base.TTY, ::Type, ::Base.OrCmds) at C:\Users\phlavenk\.julia\packages\BinaryProvider\ZbFxn\src\OutputCollector.jl:118
   [17] OutputCollector(::Base.OrCmds) at C:\Users\phlavenk\.julia\packages\BinaryProvider\ZbFxn\src\OutputCollector.jl:115
   [18] macro expansion at C:\Users\phlavenk\.julia\packages\BinaryProvider\ZbFxn\test\runtests.jl:101 [inlined]
   [19] macro expansion at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v0.7\Test\src\Test.jl:1079 [inlined]
   [20] top-level scope at C:\Users\phlavenk\.julia\packages\BinaryProvider\ZbFxn\test\runtests.jl:34
   [21] include at .\boot.jl:317 [inlined]
   [22] include_relative(::Module, ::String) at .\loading.jl:1038
   [23] include(::Module, ::String) at .\sysimg.jl:29
   [24] include(::String) at .\client.jl:398
   [25] top-level scope at none:0
   [26] eval(::Module, ::Any) at .\boot.jl:319
   [27] macro expansion at .\logging.jl:317 [inlined]
   [28] exec_options(::Base.JLOptions) at .\client.jl:229
   [29] _start() at .\client.jl:432
Test Summary:   | Pass  Error  Total
OutputCollector |   15      1     16
ERROR: LoadError: Some tests did not pass: 15 passed, 0 failed, 1 errored, 0 broken.
in expression starting at C:\Users\phlavenk\.julia\packages\BinaryProvider\ZbFxn\test\runtests.jl:33
ERROR: Package BinaryProvider errored during testing

JuliaPro MKL support: allow no-busybox mode

Example error:

INFO: Building Sundials
==============================[ ERROR: Sundials ]===============================
LoadError: InitError: No sh engines found. We looked for: C:\Users\Chris\AppData\Local\JuliaPro-0.6.2.2\Julia-0.6.2\bin\busybox.exe, sh. Install one and ensure it is available on the path.

during initialization of module BinaryProvider
while loading C:\Users\Chris\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Sundials\deps\build.jl, in expression starting on line 1

================================================================================
================================[ BUILD ERRORS ]================================
WARNING: Sundials had build errors.

 - packages with build errors remain installed in C:\Users\Chris\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6
 - build the package(s) and all dependencies with `Pkg.build("Sundials")`
 - build a single package by running its `deps/build.jl` script

================================================================================

Downloaded binaries get used no matter what

When you change the build script url, to e.g. use a new tag with new binaries, a user running Pkg.build("MyPackage") will reuse whatever binary is already downloaded
When the download is corrupted, or from an older tag with non matching hashes, build fails, even though deleting and downloading the new binary would resolve the problem
Downloading from non existent urls works as if nothing is wrong, the empty file stays in the download folder and gets used, but fails with a hash not match error

Thanks for all the effort!

Not sure best way to deal w/ shared library down one level too deep

I think it's not uncommon for a make install to create a directory structure like prefix/bin/libfoo/foo or prefix/lib/libfoo/libfoo.so (see attached screenshot for mariadb) Not sure what the best way to handle this is. The BinaryBuilder is setup to just do a regular cmake && make && make install, but then BinaryProvider has a hard time later because it's not located at usr/lib/libfoo.so. I'm not sure if BinaryProvider could be more flexible here in recursing down a bit? Or should BinaryBuilder detect this kind of structure and complain? Or rearrange things itself?

MethodError: no method matching position(::Pipe)

This error popped up just after the build retry step of the BinaryBuilder wizard. Unfortunately the backtrace just shows the point where the task is being waited for, not where the actual error occured.

It looks like this might be this issue in Julia where reading a Char from a Process throws this error, which is what's happening here. The fix seems to be merged into 1.0 but still pending backporting for 0.7. Perhaps it's worth adding a workaround?

MethodError: no method matching position(::Pipe)
Closest candidates are:
  position(::HTTP.Form) at /home/sfr/.julia/packages/HTTP/nUK4f/src/multipart.jl:23
  position(::Base.SecretBuffer) at secretbuffer.jl:154
  position(::Base.Filesystem.File) at filesystem.jl:225
  ...
Stacktrace:
 [1] _wait(::Task) at ./task.jl:192
 [2] fetch at ./task.jl:205 [inlined]
 [3] wait(::OutputCollector) at /home/sfr/.julia/packages/BinaryProvider/ArZWB/src/OutputCollector.jl:155
 [4] (::getfield(BinaryBuilder, Symbol("##78#80")){Bool,Base.TTY,BinaryBuilder.UserNSRunner,Cmd,String})() at /home/sfr/.julia/dev/BinaryBuilder/src/UserNSRunner.jl:126
 [5] cd(::getfield(BinaryBuilder, Symbol("##78#80")){Bool,Base.TTY,BinaryBuilder.UserNSRunner,Cmd,String}, ::String) at ./file.jl:96
 [6] #run#77(::Bool, ::Base.TTY, ::Function, ::BinaryBuilder.UserNSRunner, ::Cmd, ::String) at /home/sfr/.julia/dev/BinaryBuilder/src/UserNSRunner.jl:123
 [7] (::getfield(Base, Symbol("#kw##run")))(::NamedTuple{(:verbose, :tee_stream),Tuple{Bool,Base.TTY}}, ::typeof(run), ::BinaryBuilder.UserNSRunner, ::Cmd, ::String) at ./none:0
 [8] (::getfield(BinaryBuilder, Symbol("##217#218")){BinaryBuilder.WizardState,Linux,String})() at /home/sfr/.julia/dev/BinaryBuilder/src/wizard/interactive_build.jl:207
 [9] cd(::getfield(BinaryBuilder, Symbol("##217#218")){BinaryBuilder.WizardState,Linux,String}, ::String) at ./file.jl:96
 [10] step3_retry(::BinaryBuilder.WizardState) at /home/sfr/.julia/dev/BinaryBuilder/src/wizard/interactive_build.jl:192
 [11] run_wizard(::BinaryBuilder.WizardState) at /home/sfr/.julia/dev/BinaryBuilder/src/Wizard.jl:60
 [12] run_wizard() at /home/sfr/.julia/dev/BinaryBuilder/src/Wizard.jl:24

Cache probe_platform_engines!

Running probe_plastform_engines! takes about 0.6 seconds. Perhaps it is worth to cache the results as a build step? This would require rebuilding when a user reinstalls new engines but a 0.6-second startup seems a bit high for something that quite rarely changes?

I'm looking a bit into downloading packages as tarballs in Pkg3 and am currently using this package and this causes startup time to increase a bit too much. If this issue is not in line with how the package is thought to be used, I could, of course, roll my own build script.

ERROR: LoadError: LoadError: libstdc++ ABI version '3.4.0' too small; please build Julia with GCC >= 4.8.5, when Julia built with GCC 7.3

Now, I'm not confident that BinaryProvider is at fault, but I would appreciate any suggestions on how to further diagnose and fix the issue.

For reference, I built Julia with:

$ gcc --version
gcc (Ubuntu 7.3.0-16ubuntu3) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Trying to debug by running code from here leads me to:

julia> libstdcxx_paths = filter(x -> occursin("libstdc++", x), dllist())
1-element Array{AbstractString,1}:
 "/usr/lib/x86_64-linux-gnu/libstdc++.so.6"

julia> hdl = dlopen(first(libstdcxx_paths))

julia> dlsym_e(hdl, "GLIBCXX_3.4.25") # Null pointer
Ptr{Nothing} @0x0000000000000000

julia> dlsym_e(hdl, "GLIBCXX_3.4.13") # Null pointer
Ptr{Nothing} @0x0000000000000000

julia> max_version = v"3.4.0";

julia> for minor_version in 1:26
           if dlsym_e(hdl, "GLIBCXX_3.4.$(minor_version)") != C_NULL
               max_version = VersionNumber("3.4.$(minor_version)")
           end
       end

julia> max_version
v"3.4.0"

Trying to check the file from the command line...

$ nm -D /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX_3.4
0000000000000000 A GLIBCXX_3.4
0000000000000000 A GLIBCXX_3.4.1
0000000000000000 A GLIBCXX_3.4.10
0000000000000000 A GLIBCXX_3.4.11
0000000000000000 A GLIBCXX_3.4.12
0000000000000000 A GLIBCXX_3.4.13
0000000000000000 A GLIBCXX_3.4.14
0000000000000000 A GLIBCXX_3.4.15
0000000000000000 A GLIBCXX_3.4.16
0000000000000000 A GLIBCXX_3.4.17
0000000000000000 A GLIBCXX_3.4.18
0000000000000000 A GLIBCXX_3.4.19
0000000000000000 A GLIBCXX_3.4.2
0000000000000000 A GLIBCXX_3.4.20
0000000000000000 A GLIBCXX_3.4.21
0000000000000000 A GLIBCXX_3.4.22
0000000000000000 A GLIBCXX_3.4.23
0000000000000000 A GLIBCXX_3.4.24
0000000000000000 A GLIBCXX_3.4.25
0000000000000000 A GLIBCXX_3.4.3
0000000000000000 A GLIBCXX_3.4.4
0000000000000000 A GLIBCXX_3.4.5
0000000000000000 A GLIBCXX_3.4.6
0000000000000000 A GLIBCXX_3.4.7
0000000000000000 A GLIBCXX_3.4.8
0000000000000000 A GLIBCXX_3.4.9

So it contains the symbols up through 3.4.25. Although, each is also accompanied with what may be a null address on the left?
For example, if I don't grep, the last couple functions are:

000000000037bf60 V _ZTVSt9strstream
000000000037b490 V _ZTVSt9type_info

although the results are inconsistent with dlsym_e:

julia> dlsym_e(hdl, "_ZTVSt9type_info")
Ptr{Nothing} @0x00007ff0ef183490

but perhaps that's because of the "V" (weak symbol object)?

Now, I also see the lines:

    # Full list available here: https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html
    if max_version < v"3.4.18"
        error("libstdc++ ABI version '$max_version' too small; please build Julia with GCC >= 4.8.5")
    elseif max_version < v"3.4.23"
        # If we aren't up to 7.1.0, then we fall all the way back to 4.8.5
        return :gcc4
    elseif max_version < v"3.4.25"
        return :gcc7
    else
        return :gcc8
    end

which, because grep indicates it has v"3.4.25", which is not less than itself, it would be categorized as gcc8, when in fact it is gcc7, as shown above.
The link only states that v"3.4.24" is for 7.2, not 7.3.
Although, maybe that means gcc 7.3 has gcc 8's libstdcxx_abi, in which case no problem. I know almost nothing about these things, and would appreciate clarification.

Where are the errors?
Should the addresses of the symbols listed by nm not be 0? Is something wrong with my libstdc++.so.6?
Or, is the null pointer check for them not a valid check to determine which version?

This was also a rather confusing error message to track down, because I had not built Julia with a gcc <= 4.8.5.

I reproduced this problem on another computer.

option to specify endianness of download

I know that Julia has been little-endian focused, and I agree with that. Now that people are starting to work with specialty data and compressed files generated with specialized hardware, allowing the user of this to state by keyword setting that the target (file or compressed collection of files) has been stored big-endian.
(I have no such need).

Tag?

Sundials.jl works with the master branch of BinaryProvider. I was wondering if we could get a tag to really try this out. Thanks!

Allow `busybox` as download/compress engine

Busybox in single-executable mode, can be called as busybox wget and busybox tar. If busybox is available, we should be able to use it like that, even if the individual symlinks are not available.

spawn not defined

│ ERROR: LoadError: LoadError: LoadError: LoadError: UndefVarError: spawn not defined
juliav1.0.0rc1.0

Build "julia shipped libraries" package

With each Julia release, we should generate a BinaryProvider-style tarball of shared libraries that typically ship with Julia. Things like libgfortran.so.4, libstdc++.so.6, etc... and enable automatic downloading of it to the julia private shlib location. These will then override the typical system libraries, and since they will be newer (and backwards-compatible) they should enable from-source builders of Julia to use the precompiled binaries that we ship through BinaryBuilder.

We will have to update this package of libraries every time the compilers we're using change something (a la libgfortran.so.3 -> libgfortran.so.4 in GCC 7) but hopefully we can keep those problems limited to a particular Julia major release.

no silent error when locating libraries (enhancement)

I am trying to install EzXML, but the installation process failed with the error:

 Error: Error building `EzXML`: 
│ ERROR: LoadError: LibraryProduct(nothing, ["libxml2"], :libxml2, "Prefix(/home/ulg/gher/abarth/.julia/packages/EzXML/3U9Ow/deps/usr)") is not satisfied, cannot generate deps.jl!

The problem was that libxml2.so require libz, but the system libz is too old.
It would be great if a user would see the error message of dlopen, instead of failing silently.

https://github.com/JuliaPackaging/BinaryProvider.jl/blob/master/src/Products.jl#L172

juliapackaging / binaryprovider.jl Goto Github PK

binaryprovider.jl's People

Contributors

Stargazers

Watchers

Forkers

binaryprovider.jl's Issues

Recommend Projects

Recommend Topics

Recommend Org