Giter VIP home page Giter VIP logo

terminus_store_prolog's Introduction

TerminusDB Logo

CI Issues

TerminusDB is a distributed database with a collaboration model.

It is designed to be like git, but for data. If you are reading this, give this repo a star.

The building blocks of the model are -

  • Revision Control: commits for every update
  • Diff: differences between commits can be interpreted as patches between states
  • Push/Pull/Clone: communicate diffs between nodes using push / pull / clone
  • Query: You can query any state of the database at any commit.

TerminusDB allows you to link JSON documents in a knowledge graph through a document API. TerminusDB is available as a standalone server, or you can use our headless content and knowledge management system TerminusCMS.

TerminusDB Version 11

TerminusDB 11 features a new Rustified storage backend that reduces storage overhead and latency, improves search performance, and simplifies interchange. TerminusDB 11 also comes with some exciting features to make building easier and faster -

  • GraphQL - Use GraphQL as a proper graph query language with deep link discover and path queries
  • Added @unfoldable document flag to frames - Making data curation easier by unfolding subdocuments within a frame to add all relevant data in one place
  • Add @metadata to frames - Include additional metadata to document frames including data formatted as Markdown.

Installation Guide

Get it from the Snap Store

The easiest way to install TerminusDB as a developer is by using the Snap. It does not provide a daemon and is mainly intended for developers that want to try TerminusDB.

For deployments, or if you want to use the local dashboard:

  1. Add the following to a .env file in the source directory:
OPENAI_KEY=YOUR_OPENAI_KEY_HERE
# And optionally specify number of pages for the vector database
# for instance
BUFFER_AMOUNT=120000
  1. docker compose up

You should be able to view TerminusDB running by default at localhost:6363

If you're installing TerminusDB on Windows with Docker, our friends at DFRNT wrote this comprehensive guide.

You can also install TerminusDB from the Source Code.

TerminusDB CLI

A simple example creating a person with friends can be created as follows:

terminusdb db create admin/example1
terminusdb doc insert --graph_type=schema admin/example1 <<EOF
{ "@id" : "Person",
  "@type" : "Class",
  "name" : "xsd:string",
  "occupation" : "xsd:string",
  "friends" : { "@type" : "Set",
                "@class" : "Person" }}
EOF
terminusdb doc insert admin/example1 --message='adding Gavin' <<EOF
{ "@type" : "Person","name" : "Gavin", "occupation" : "Coder"}
EOF

You can attach this to a database on TerminusCMS in order to distribute it by setting your remote. First you will need to create a resource from TerminusCMS. This is analogous to creating a repository on GitHub.

Log into TerminusCMS, create a new data product, and make sure you have an access token to the team in which we created that data product. Then copy the URL to clone from the data product info page.

If you make a data product called example in the team Terminators. We could then issue the following command using the TerminusDB CLI. You should get an API token from your profile page and replace 'XYZ' with this token.

terminusdb remote add admin/example1 origin 'https://cloud-dev.terminusdb.com/Terminators/example' --token='XYZ'
terminusdb fetch admin/example1 --token='XYZ'
terminusdb push admin/example1 --token='XYZ'

Once completed, you'll have a remote copy of this database.

More on the CLI here

Community

Come visit us on Discord or our community Subreddit. On Twitter, we're @TerminusDB.

Documentation

Check out our documentation site for more information: TerminusDB documentation website.

We are working hard to improve our docs - if you see an issue, please open an issue in the documentation repo.

White paper on our delta-encoding approach to data management.

Check the Python Client ๐Ÿ. Or the JavaScript Client ๐ŸŒ.

WOQL is a powerful query language that allows you to express complex patterns over arbitrary data structures concisely. What makes it so expressive and easy to use is the radical simplicity of the core underlying concepts.

Changes in this Version

Release Notes

Copyright

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

terminus_store_prolog's People

Contributors

anniepoo avatar dnknth avatar gavinmendelgleason avatar matko avatar pmoura avatar rrooij avatar spl avatar zeroloss avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

terminus_store_prolog's Issues

terminus_store:subject_predicate_lookup_object/2 leaves choice point.

  1. run the terminus_store_test
swipl run_test.pl -- -n 1 
?- tspy(opt_test).
?-go.

Graphic debugger will stop at opt_test(one_big_union_er_layer)

s to step over

EXPECTED: last item on stack is opt_test
OBSERVED: 10_000 choice points for terminus_store:subject_predicate_lookup_object/2

improve bitmap indexing

Bitmap indexing was written in a rush, and there's some TODO's in that code regarding how to pick a proper logarray width.

create_or_open_named_graph

It would be convenient to have a create_or_open_named_graph instead of having to somehow know if the graph exists and call either create or open.

Readme example not working

Trying the first example in the readme file after successfully installing the pack:

?- use_module(library(terminus_store)).
true.

?- open_directory_store("testdir", Store),
|    open_write(Store, Builder),
|    open_named_graph(Store, "sometestdb", DB),
|    nb_add_triple(Builder, "Subject", "Predicate", value("Object")),
|    nb_commit(Builder, Layer),
|    nb_set_head(DB, Layer).
ERROR: Unhandled exception: terminus_store_rust_error('No such file or directory (os error 2)')

Tracing the query gives:

   Call: (11) terminus_store:open_directory_store("testdir", _5696) ? creep
   Exit: (11) terminus_store:open_directory_store("testdir", <store_blob>) ? creep
   Call: (11) terminus_store:open_write(<store_blob>, _5702) ? creep
ERROR: Unhandled exception: terminus_store_rust_error('No such file or directory (os error 2)')
   Exception: (11) terminus_store:open_write(<store_blob>, _5702) ? 

Pack cannot be installed on macOS

Trying to install the pack in macOS (running SWI-Prolog 8.1.16) fails with the error:

%    Compiling tokio-uds v0.2.5
%    Compiling tokio v0.1.22
%    Compiling terminus-store v0.7.6
%    Compiling terminus-store-prolog v0.1.0 (/Users/pmoura/.local/share/swi-prolog/pack/terminus_store_prolog/rust)
%     Finished release [optimized] target(s) in 1m 08s
% cp rust/target/release//libterminus_store_prolog.so  lib/x86_64-darwin
% cp: rust/target/release//libterminus_store_prolog.so: No such file or directory
% make: *** [build] Error 1

The error results from the Rust compilation generates a .dylib file instead of a .so file:

$ ll /Users/pmoura/.local/share/swi-prolog/pack/terminus_store_prolog/rust/target/release/
total 5904
drwxr-xr-x   22 pmoura  staff      704 Nov 22 15:10 build
drwxr-xr-x  160 pmoura  staff     5120 Nov 22 15:11 deps
drwxr-xr-x    2 pmoura  staff       64 Nov 22 15:10 examples
drwxr-xr-x    2 pmoura  staff       64 Nov 22 15:10 incremental
-rw-r--r--    1 pmoura  staff      198 Nov 22 15:11 libterminus_store_prolog.d
-rwxr-xr-x    2 pmoura  staff  3017552 Nov 22 15:11 libterminus_store_prolog.dylib
drwxr-xr-x    2 pmoura  staff       64 Nov 22 15:10 native

The .so extension is specified in the `Makefile:

$ cat /Users/pmoura/.local/share/swi-prolog/pack/terminus_store_prolog/Makefile 
RUST_LIB_NAME = terminus_store_prolog
RUST_LIB = lib$(RUST_LIB_NAME).$(SOEXT)
RUST_TARGET=release
RUST_TARGET_DIR = rust/target/$(RUST_TARGET)/
SOEXT = so 

But the SOEXT value is taken from the buildenv.sh file generated by the query prolog_pack:save_build_environment('./'). I don't know why SWI-Prolog is setting the extension to .so in macOS where the default for shared libraries is .dylib. Thus, it's not clear where this issue should be fixed.

pack_install fails

SWI-Prolog version 8.4.1 for arm64-darwin
cargo 1.57.0 (b2e52d7ca 2021-10-21)

Also fails to build locally with the same error.

...

%   Compiling swipl-macros v0.3.3
%    Compiling thiserror v1.0.30
%    Compiling futures-executor v0.3.17
%    Compiling futures v0.3.17
%    Compiling swipl-fli v0.3.2
%    Compiling futures-locks v0.6.0
%    Compiling tokio-util v0.6.8
%    Compiling terminus-store v0.19.5
%    Compiling swipl v0.3.5
%    Compiling terminus-store-prolog v0.2.0 (/Users/avanti/.local/share/swi-prolog/pack/terminus_store_prolog/rust)
%     Finished release [optimized] target(s) in 40.34s
% cp rust/target/release/libterminus_store_prolog.so \
% 	   lib/arm64-darwin/libterminus_store.so
% cp: rust/target/release/libterminus_store_prolog.so: No such file or directory
% make: *** [build] Error 1
ERROR: Process "process(path(make),[all])": exit status: 2
ERROR: In:
ERROR:   [21] throw(error(process_error(...,...),_11834))
ERROR:   [20] prolog_pack:run_process(path(make),[all],[directory('/Users/avanti/.local/share/swi-prolog/pack/terminus_store_prolog'),...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:2333
ERROR:   [19] forall(prolog_pack:member(all,...),prolog_pack:run_process(...,...,...)) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/boot/apply.pl:52
ERROR:   [18] prolog_pack:pack_make('/Users/avanti/.local/share/swi-prolog/pack/terminus_store_prolog',[all,check|...],[inquiry(true),...|...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:1252
ERROR:   [17] prolog_pack:make_foreign('/Users/avanti/.local/share/swi-prolog/pack/terminus_store_prolog',[inquiry(true),...|...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:1231
ERROR:   [16] prolog_pack:post_install_foreign(terminus_store_prolog,'/Users/avanti/.local/share/swi-prolog/pack/terminus_store_prolog',[inquiry(true),...|...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:1116
ERROR:   [15] prolog_pack:pack_post_install(terminus_store_prolog,'/Users/avanti/.local/share/swi-prolog/pack/terminus_store_prolog',[inquiry(true),...|...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:1077
ERROR:   [14] prolog_pack:pack_install_from_local('/Users/avanti/.local/share/swi-prolog/pack/Downloads/terminus_store_prolog-0.19.5.zip','/Users/avanti/.local/share/swi-prolog/pack',terminus_store_prolog,[inquiry(true),...|...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:768
ERROR:   [13] prolog_pack:pack_install_from_url(https,'https://github.com/terminusdb/terminus_store_prolog/archive/v0.19.5.zip','/Users/avanti/.local/share/swi-prolog/pack',terminus_store_prolog,[inquiry(true),...|...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:995
ERROR:   [12] prolog_pack:pack_install(terminus_store_prolog,'/Users/avanti/.local/share/swi-prolog/pack',[inquiry(true),...|...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:738
ERROR:   [11] prolog_pack:pack_install(terminus_store_prolog,[pack(terminus_store_prolog),...|...]) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:653
ERROR:   [10] prolog_pack:pack_install(terminus_store_prolog) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/library/prolog_pack.pl:475
ERROR:    [9] toplevel_call(user:user: ...) at /opt/homebrew/Cellar/swi-prolog/8.4.1/libexec/lib/swipl/boot/toplevel.pl:1117

implement nondet query predicates without creating lookup blobs

In the current implementation, doing id_triple($Layer, S, P, O) will call various nondeterministic foreign predicates, all creating blobs:

id_triple(Layer, Subject, Predicate, Object) :-
    lookup_subject(Layer, Subject_Lookup),
    subject_lookup_subject(Subject_Lookup, Subject),
    subject_lookup_predicate(Subject_Lookup, Predicate_Lookup),
    subject_predicate_lookup_predicate(Predicate_Lookup, Predicate),
    subject_predicate_lookup_object(Predicate_Lookup, Object).

Here, lookup_subject/2 and subject_lookup_predicate/2 are both nondet predicates which bind a lookup blob to their second argument, leaving a choicepoint to bind a different on on backtrack.

This means that just iterating over a dataset will create a lot of lookup blobs (as many as distinct_subjects + distinct_subject_predicate_pairs), all of which are almost instantly forgotten and left for the garbage collector to collect.

Lookups are nice in theory, cause it allows us to implement more complex lookup patterns, where we can cache parts of the lookup. For the basic triple lookup though, they are inefficient. Much better would be for the iteration to happen entirely on the native side, which should not return lookup blobs but rather bind the subject/predicate/object id directly.

Pack cannot be installed in Ubuntu 18

Trying to install the pack (using the pack_install(terminus_store_prolog) query on Ubuntu 18.04.3 using cargo and rustc 1.36.0 fails with the compilation error:

%    Compiling terminus-store v0.7.6
ERROR: error: enum variants on type aliases are experimental
ERROR:    --> /home/pmoura/.cargo/registry/src/github.com-1ecc6299db9ec823/terminus-store-0.7.6/src/layer/layer.rs:459:13
ERROR:     |
ERROR: 459 |             Self::Unresolved(_) => false,
ERROR:     |             ^^^^^^^^^^^^^^^^^^^

open_directory_store when directory does not exist

We should check for directory existance when opening the store.

On a machine WITHOUT a /tmp/demo directory:

Welcome to SWI-Prolog (threaded, 64 bits, version 8.0.3)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit http://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- open_directory_store('/tmp/demo', Store).
Store = <store_blob> .

?- open_write($Store, Builder).
ERROR: Unhandled exception: terminus_store_rust_error('No such file or directory (os error 2)')

EXPECTED: throw exception from open_directory_store/2
OBSERVED: lazily only flips when the store's written

calling open_write more than once workflow is baffling

The current scheme with builder and layer and getting a builder is pretty confusing. Perhaps head should be guaranteed to return some layer or something - the open_write the first time on the graph, and from then on on the layer was baffling.

Pack won't install/compile on macOS Apple Silicon

The $SWIARCH test in the Makefile only matches on x86_64-darwin, so it tries to build using the default (Linux) $BUILD_LD_OPTIONS. These options aren't understood by the XCode linker, so the build fails.

I tried adding an arm64-darwin test arm that uses the same $BUILD_LD_OPTIONS; that build succeeds and passes the tests.

I'm not sure whether the "correct" fix would be to switch on $SWIARCH or $OS, though, since I don't know the reasons why the Darwin test is architecture-specific.

Error in README.md

The section Convert strings to ids and query by id uses old artity 2 predicates for subject_id, predicate_id, and object_id.

It works if changed to:

open_directory_store("testdir", Store),
open_named_graph(Store, "sometestdb", DB),
head(DB, Layer),
subject_id(Layer, "Subject", S_Id),
id_triple(Layer, S_Id, P_Id, O_Id),
predicate_id(Layer, Predicate, P_Id),
object_id(Layer, Object, O_Id).

Implement label deletion

Label deletion is implemented in the store library, but not yet in the prolog library. Make it so.

Non-necessary use proprietary compound term with zero arguments in clause head?

In the definition of the following predicate:

is there any reason to use a proprietary compound term with zero arguments in clause head instead of an atom? All calls to this predicate don't seem to require it.

Found this issue while debugging some odd exceptions that occur with SWI-Prolog 8.0.3 but not with 8.1.16, which suggests that clause head is triggering a SWI-Prolog bug in the stable version that's already fixed in the current devel version.

resolve the hard coded shared lib mess

Currently we use some bash script wrapper to get the default directories correct so all the shared libs load. This is ugly and incorrect and we should fix this. I started writing a load tester and quickly discovered it's painful.

Anyway, it doesn't seem 'what should be done'.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.