terminusdb / terminus_store_prolog Goto Github PK

View Code? Open in Web Editor NEW

20.0 8.0 6.0 446 KB

Prolog binding for terminusdb-store

License: Apache License 2.0

Makefile 1.27% Prolog 48.85% Rust 46.23% Shell 3.03% Dockerfile 0.62%

terminus_store_prolog's Introduction

terminus-store prolog bindings

Prolog bindings for the terminus-store Rust library.

Archival notice

This project has been archived. Its primary user, TerminusDB, has directly integrated the code of this repository, and all development is now happening there.

Requirements

cargo
clang
swi-prolog (with the include headers)

Installing

This library is downloadable through SWI-Prolog's package management system. In a swipl instance, run

pack_install(terminus_store_prolog).

Then you can use the library with

use_module(library(terminus_store)).

Compiling and running without installing (for testing purposes)

If you need to compile manually, for example to test a change without reinstalling the pack, follow these instructions.

Also, use the provided ./script/swipl script to start a test instance. This will ensure the foreign library will be located properly.

make
./script/swipl

Running the tests

make
./script/test

Examples

Creating a named graph and adding a triple

Create a new directory (testdir in this example), then do the following:

open_directory_store("testdir", Store),
open_write(Store, Builder),
create_named_graph(Store, "sometestdb", DB),
nb_add_triple(Builder, "Subject", "Predicate", value("Object")),
nb_commit(Builder, Layer),
nb_set_head(DB, Layer).

Add a triple to an existing named graph

open_directory_store("testdir", Store),
open_named_graph(Store, "sometestdb", DB),
open_write(DB, Builder),
nb_add_triple(Builder, "Subject2", "Predicate2", value("Object2")),
nb_commit(Builder, Layer),
nb_set_head(DB, Layer),

Query triples

open_directory_store("testdir", Store),
open_named_graph(Store, "sometestdb", DB),
head(DB, Layer),
triple(Layer, Subject, Predicate, Object).

Convert strings to ids and query by id

open_directory_store("testdir", Store),
open_named_graph(Store, "sometestdb", DB),
head(DB, Layer),
subject_id(Layer, "Subject", S_Id),
id_triple(Layer, S_Id, P_Id, O_Id),
predicate_id(Layer, Predicate, P_Id),
object_id(Layer, Object, O_Id).

terminus_store_prolog's People

Contributors

Stargazers

Watchers

Forkers

pmoura codiepp curtosis spl kaii-zen hierophantos

terminus_store_prolog's Issues

Pack cannot be installed on macOS

Trying to install the pack in macOS (running SWI-Prolog 8.1.16) fails with the error:

%    Compiling tokio-uds v0.2.5
%    Compiling tokio v0.1.22
%    Compiling terminus-store v0.7.6
%    Compiling terminus-store-prolog v0.1.0 (/Users/pmoura/.local/share/swi-prolog/pack/terminus_store_prolog/rust)
%     Finished release [optimized] target(s) in 1m 08s
% cp rust/target/release//libterminus_store_prolog.so  lib/x86_64-darwin
% cp: rust/target/release//libterminus_store_prolog.so: No such file or directory
% make: *** [build] Error 1

The error results from the Rust compilation generates a .dylib file instead of a .so file:

$ ll /Users/pmoura/.local/share/swi-prolog/pack/terminus_store_prolog/rust/target/release/
total 5904
drwxr-xr-x   22 pmoura  staff      704 Nov 22 15:10 build
drwxr-xr-x  160 pmoura  staff     5120 Nov 22 15:11 deps
drwxr-xr-x    2 pmoura  staff       64 Nov 22 15:10 examples
drwxr-xr-x    2 pmoura  staff       64 Nov 22 15:10 incremental
-rw-r--r--    1 pmoura  staff      198 Nov 22 15:11 libterminus_store_prolog.d
-rwxr-xr-x    2 pmoura  staff  3017552 Nov 22 15:11 libterminus_store_prolog.dylib
drwxr-xr-x    2 pmoura  staff       64 Nov 22 15:10 native

The .so extension is specified in the `Makefile:

$ cat /Users/pmoura/.local/share/swi-prolog/pack/terminus_store_prolog/Makefile 
RUST_LIB_NAME = terminus_store_prolog
RUST_LIB = lib$(RUST_LIB_NAME).$(SOEXT)
RUST_TARGET=release
RUST_TARGET_DIR = rust/target/$(RUST_TARGET)/
SOEXT = so

But the SOEXT value is taken from the buildenv.sh file generated by the query prolog_pack:save_build_environment('./'). I don't know why SWI-Prolog is setting the extension to .so in macOS where the default for shared libraries is .dylib. Thus, it's not clear where this issue should be fixed.

terminus_store:subject_predicate_lookup_object/2 leaves choice point.

run the terminus_store_test

swipl run_test.pl -- -n 1 
?- tspy(opt_test).
?-go.

Graphic debugger will stop at opt_test(one_big_union_er_layer)

s to step over

EXPECTED: last item on stack is opt_test
OBSERVED: 10_000 choice points for terminus_store:subject_predicate_lookup_object/2

Archive terminus_store_prolog

We have integrated terminus_store_prolog into the terminusdb repository (terminusdb/terminusdb#1160), so we should leave a note on the README.md and archive the terminus_store_prolog repository.

Bare labels not being type checked in schema checking phase

Schema checking needs to check to see that comment and label fields are well typed before allowing them to be entered.

calling open_write more than once workflow is baffling

The current scheme with builder and layer and getting a builder is pretty confusing. Perhaps head should be guaranteed to return some layer or something - the open_write the first time on the graph, and from then on on the layer was baffling.

Pack cannot be installed in Ubuntu 18

Trying to install the pack (using the pack_install(terminus_store_prolog) query on Ubuntu 18.04.3 using cargo and rustc 1.36.0 fails with the compilation error:

%    Compiling terminus-store v0.7.6
ERROR: error: enum variants on type aliases are experimental
ERROR:    --> /home/pmoura/.cargo/registry/src/github.com-1ecc6299db9ec823/terminus-store-0.7.6/src/layer/layer.rs:459:13
ERROR:     |
ERROR: 459 |             Self::Unresolved(_) => false,
ERROR:     |             ^^^^^^^^^^^^^^^^^^^

Pack won't install/compile on macOS Apple Silicon

The $SWIARCH test in the Makefile only matches on x86_64-darwin, so it tries to build using the default (Linux) $BUILD_LD_OPTIONS. These options aren't understood by the XCode linker, so the build fails.

I tried adding an arm64-darwin test arm that uses the same $BUILD_LD_OPTIONS; that build succeeds and passes the tests.

I'm not sure whether the "correct" fix would be to switch on $SWIARCH or $OS, though, since I don't know the reasons why the Darwin test is architecture-specific.

make `nb_remove_triple` work with non-ground parameters

Right now, every parameter to nb_remove_triple/4 needs to be ground. For mass removal, it'd be useful if parameters could be non-ground, meaning any matching triple is to be deleted.

values must be atoms or strings

people will want to store ints and floats and such

Readme example not working

Trying the first example in the readme file after successfully installing the pack:

?- use_module(library(terminus_store)).
true.

?- open_directory_store("testdir", Store),
|    open_write(Store, Builder),
|    open_named_graph(Store, "sometestdb", DB),
|    nb_add_triple(Builder, "Subject", "Predicate", value("Object")),
|    nb_commit(Builder, Layer),
|    nb_set_head(DB, Layer).
ERROR: Unhandled exception: terminus_store_rust_error('No such file or directory (os error 2)')

Tracing the query gives:

   Call: (11) terminus_store:open_directory_store("testdir", _5696) ? creep
   Exit: (11) terminus_store:open_directory_store("testdir", <store_blob>) ? creep
   Call: (11) terminus_store:open_write(<store_blob>, _5702) ? creep
ERROR: Unhandled exception: terminus_store_rust_error('No such file or directory (os error 2)')
   Exception: (11) terminus_store:open_write(<store_blob>, _5702) ?

resolve the hard coded shared lib mess

Currently we use some bash script wrapper to get the default directories correct so all the shared libs load. This is ugly and incorrect and we should fix this. I started writing a load tester and quickly discovered it's painful.

Anyway, it doesn't seem 'what should be done'.

Non-necessary use proprietary compound term with zero arguments in clause head?

In the definition of the following predicate:

terminus_store_prolog/prolog/terminus_store.pl

Line 416 in 98599a9

createng() :-

is there any reason to use a proprietary compound term with zero arguments in clause head instead of an atom? All calls to this predicate don't seem to require it.

Found this issue while debugging some odd exceptions that occur with SWI-Prolog 8.0.3 but not with 8.1.16, which suggests that clause head is triggering a SWI-Prolog bug in the stable version that's already fixed in the current devel version.

open_directory_store when directory does not exist

We should check for directory existance when opening the store.

On a machine WITHOUT a /tmp/demo directory:

Welcome to SWI-Prolog (threaded, 64 bits, version 8.0.3)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit http://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- open_directory_store('/tmp/demo', Store).
Store = <store_blob> .

?- open_write($Store, Builder).
ERROR: Unhandled exception: terminus_store_rust_error('No such file or directory (os error 2)')

EXPECTED: throw exception from open_directory_store/2
OBSERVED: lazily only flips when the store's written

pack_install fails

SWI-Prolog version 8.4.1 for arm64-darwin
cargo 1.57.0 (b2e52d7ca 2021-10-21)

Also fails to build locally with the same error.

...

%   Compiling swipl-macros v0.3.3
%    Compiling thiserror v1.0.30
%    Compiling futures-executor v0.3.17
%    Compiling futures v0.3.17
%    Compiling swipl-fli v0.3.2
%    Compiling futures-locks v0.6.0
%    Compiling tokio-util v0.6.8
%    Compiling terminus-store v0.19.5
%    Compiling swipl v0.3.5
%    Compiling terminus-store-prolog v0.2.0 (/Users/avanti/.local/share/swi-prolog/pack/terminus_store_prolog/rust)
%     Finished release [optimized] target(s) in 40.34s
% cp rust/target/release/libterminus_store_prolog.so \
% 	   lib/arm64-darwin/libterminus_store.so
% cp: rust/target/release/libterminus_store_prolog.so: No such file or directory
% make: *** [build] Error 1
ERROR: Process "process(path(make),[all])": exit status: 2
ERROR: In:
ERROR:   [21] throw(error(process_error(...,...),_11834))
ERROR:   [20] prolog_pack:run_process(path(make),[all],[directory('/Users/avanti/.local/share/swi-prolog/pack/terminus_store_prolog'),...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:2333
ERROR:   [19] forall(prolog_pack:member(all,...),prolog_pack:run_process(...,...,...)) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/boot/apply.pl:52
ERROR:   [18] prolog_pack:pack_make('/Users/avanti/.local/share/swi-prolog/pack/terminus_store_prolog',[all,check|...],[inquiry(true),...|...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:1252
ERROR:   [17] prolog_pack:make_foreign('/Users/avanti/.local/share/swi-prolog/pack/terminus_store_prolog',[inquiry(true),...|...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:1231
ERROR:   [16] prolog_pack:post_install_foreign(terminus_store_prolog,'/Users/avanti/.local/share/swi-prolog/pack/terminus_store_prolog',[inquiry(true),...|...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:1116
ERROR:   [15] prolog_pack:pack_post_install(terminus_store_prolog,'/Users/avanti/.local/share/swi-prolog/pack/terminus_store_prolog',[inquiry(true),...|...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:1077
ERROR:   [14] prolog_pack:pack_install_from_local('/Users/avanti/.local/share/swi-prolog/pack/Downloads/terminus_store_prolog-0.19.5.zip','/Users/avanti/.local/share/swi-prolog/pack',terminus_store_prolog,[inquiry(true),...|...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:768
ERROR:   [13] prolog_pack:pack_install_from_url(https,'https://github.com/terminusdb/terminus_store_prolog/archive/v0.19.5.zip','/Users/avanti/.local/share/swi-prolog/pack',terminus_store_prolog,[inquiry(true),...|...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:995
ERROR:   [12] prolog_pack:pack_install(terminus_store_prolog,'/Users/avanti/.local/share/swi-prolog/pack',[inquiry(true),...|...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:738
ERROR:   [11] prolog_pack:pack_install(terminus_store_prolog,[pack(terminus_store_prolog),...|...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:653
ERROR:   [10] prolog_pack:pack_install(terminus_store_prolog) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:475
ERROR:    [9] toplevel_call(user:user: ...) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/boot/toplevel.pl:1117

Implement label deletion

Label deletion is implemented in the store library, but not yet in the prolog library. Make it so.

Error in README.md

The section Convert strings to ids and query by id uses old artity 2 predicates for subject_id, predicate_id, and object_id.

It works if changed to:

open_directory_store("testdir", Store),
open_named_graph(Store, "sometestdb", DB),
head(DB, Layer),
subject_id(Layer, "Subject", S_Id),
id_triple(Layer, S_Id, P_Id, O_Id),
predicate_id(Layer, Predicate, P_Id),
object_id(Layer, Object, O_Id).

add predicates for retrieving the label list

The rust library can return a label list, but this is not yet exposed in the prolog library. Someone should add it.

create_or_open_named_graph

It would be convenient to have a create_or_open_named_graph instead of having to somehow know if the graph exists and call either create or open.

improve bitmap indexing

Bitmap indexing was written in a rush, and there's some TODO's in that code regarding how to pick a proper logarray width.

implement nondet query predicates without creating lookup blobs

In the current implementation, doing id_triple($Layer, S, P, O) will call various nondeterministic foreign predicates, all creating blobs:

id_triple(Layer, Subject, Predicate, Object) :-
    lookup_subject(Layer, Subject_Lookup),
    subject_lookup_subject(Subject_Lookup, Subject),
    subject_lookup_predicate(Subject_Lookup, Predicate_Lookup),
    subject_predicate_lookup_predicate(Predicate_Lookup, Predicate),
    subject_predicate_lookup_object(Predicate_Lookup, Object).

Here, lookup_subject/2 and subject_lookup_predicate/2 are both nondet predicates which bind a lookup blob to their second argument, leaving a choicepoint to bind a different on on backtrack.

This means that just iterating over a dataset will create a lot of lookup blobs (as many as distinct_subjects + distinct_subject_predicate_pairs), all of which are almost instantly forgotten and left for the garbage collector to collect.

Lookups are nice in theory, cause it allows us to implement more complex lookup patterns, where we can cache parts of the lookup. For the basic triple lookup though, they are inefficient. Much better would be for the iteration to happen entirely on the native side, which should not return lookup blobs but rather bind the subject/predicate/object id directly.