resolve the hard coded shared lib mess about terminus_store_prolog HOT 12 CLOSED

Yeah, it is definitely ugly, but the problem is a bit tricky because we have a stub dyn lib referring to another dyn lib. The simplest method is to hardcode the path. If anyone can make it work with out this, we should definitely go with it.

from terminus_store_prolog.

Installing as a pack should work as well. the wrapper script doesn't need to be called in that case.

But I agree that the rpath mangling feels ugly and wrong and something should be done about it.

from terminus_store_prolog.

But I agree that the rpath mangling feels ugly and wrong and something should be done about it.

Since i mentioned this a bit out of the blue I should probably clarify that we discussed this issue offline and the rpath mangling came up there.

Responding to the actual issue, basically we support 2 kinds of running terminus-store in prolog.
The main way is through the swipl pack system. The pack system takes care of setting up all relevant paths so we don't have to.
The second way is through run_swipl. This'll set up the dynamic library loading so that it overrides swipl's pack loading, before consulting the prolog code, bypassing the pack system. This allows us to do test runs without affecting the library that is installed. The foreign library loading setup is required because normally swipl will only look in a handful of places for shared objects, one of which is in the lib/<arch>/ dir of packs. Without telling swipl to first look in our build directory, it'd either find nothing, or find the shared objects of the installed version.

Similarly, there's two ways of building this library.
The first way is from within swipl through pack_install. swipl will copy the pack files to some directory it likes, then set up a proper build environment and call make (as in the command make).
The second way is by running the ./make script (so not make the command (sorry)). This calls some internal function from the pack library to replicate swipl's build environment, then call make (the command). This script is very useful for doing incremental rebuilds of the native code in an installed pack. We can't just make directly unfortunately because we need swipl's very specific build environment. As a fun aside, doing this while swipl is running causes a crash, which is why it is sometimes useful to be able to have multiple swipl instances run different versions of the library.

Finally, the rpath stuff. -rpath is a flag that goes into ld, the linker. It adds a directory path into the shared object which the dynamic object loader will look into while resolving additional shared objects. We use it to ensure that when loading the main shared object of the library (which is our c glue) it'll know where to look for the rest by hardcoding the absolute path of the rust library. This is Not Good.
What we should be doing is somehow telling swipl to look for our secondary shared object files in the same place as the primary one. Or maybe we should link them together in one big shared object. I'm not really sure.

from terminus_store_prolog.

so we have it, here's Gavin's fix

#2

from terminus_store_prolog.

My usual method of working on a pack -
Install the pack like a normal user
?- absolute_file_name(library(mypack), X).
this tells you where the pack is
go there and move the pack out of the way and instead make a symlink to your cloned directory

This usually makes all the paths happy.

from terminus_store_prolog.

I found that you can actually skip installing the pack and go straight to symlinking.

from terminus_store_prolog.

so we have it, here's Gavin's fix

#2

I'm confused, what is 'it' here?

from terminus_store_prolog.

from terminus_store_prolog.

Sorry, that should have been "Gavin made a script that fixes this - Robin pointed out the commit where he did that. I have added a link to the relevant commit."
Hence 'it' is the commit with Gavin's fix.

from terminus_store_prolog.

But actually, Gavin's changes didn't 'fix' what the issue is about (weird default directory setup in the './run_swipl' script). You'll find that removing the rpath stuff from the makefile will break both run scenarios I described above (both installing as a pack, and running through './run_swipl').

The problem here is that use_foreign_library(foreign(libterminus_store)) will look for libterminus_store in all pack foreign library locations (or, for ./run_swipl, the location we set manually) - but it won't look there for any dependent shared objects. terminus_store_prolog consists of two shared objects, the c glue and the rust code. Swipl happily picks up the c glue, but since it doesn't modify the load path while loading it, the load system can then not find the rust object. That is why we had to hardcode the path to the secondary object file in the primary object file. This is horrible. What should really happen is that swipl should be modifying the load path so that if we load a library from one place, it'll also pick up any secondary object files that might be there.

from terminus_store_prolog.

This definitely should get discussed tuesday

from terminus_store_prolog.

We don't use rpath anymore. 😄

from terminus_store_prolog.