primalhq / primal-caching-service Goto Github PK

Primal’s caching service for Nostr connects to the specified set of relays, collects all events in real time, stores them locally, and makes them available to nostr clients through a web socket-based API.

Home Page: https://primal.net

License: MIT License

Makefile 0.03% C 0.74% Nix 0.32% Julia 98.42% Vim Script 0.38% Shell 0.11%

caching decentralization infrastructure nostr scaling

primal-caching-service's People

Contributors

Stargazers

Watchers

Forkers

bitkarrot brugeman 22388o argent-oxidum ghobs91 lightningames tosunkaya noubre alexgleason rabble devdoshi abhay-raizada

primal-caching-service's Issues

Failed to precompile HTTP

This is an existing instance that I'm trying to update.

Things I've tried:

git submodule update --recursive --remote
rm -rf ~/.julia
nix develop -c sh -c '$start_primal_caching_service'
rm -rf ~/.julia
nix --extra-experimental-features 'nix-command flakes' run github:NixOS/nixpkgs/23.11#nix -- develop -c sh -c '$start_primal_caching_service'

However, this works on a new directory on the same machine:

git clone --recurse-submodules https://github.com/PrimalHQ/primal-caching-service
cd primal-caching-service/
nix develop -c sh -c '$start_primal_caching_service'

ENV:

uname -a
Linux localhost 5.10.0-25-amd64 #1 SMP Debian 5.10.191-1 (2023-08-16) x86_64 GNU/Linux

  Downloaded artifact: OpenSSL
  Downloaded artifact: SQLite
  Downloaded artifact: Libiconv
  Downloaded artifact: OpenSpecFun
Precompiling project...
  ✗ HTTP
  83 dependencies successfully precompiled in 101 seconds
  1 dependency errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the package
Precompiling project...
  ✗ HTTP
  0 dependencies successfully precompiled in 2 seconds. 83 already precompiled.

ERROR: LoadError: The following 1 direct dependency failed to precompile:

HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3]

Failed to precompile HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3] to /home/primal/.julia/compiled/v1.8/HTTP/jl_ovAiyJ.
ERROR: LoadError: ArgumentError: Package HTTP does not have ExceptionUnwrapping in its dependencies:
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have HTTP checked out for development and have
  added ExceptionUnwrapping as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with HTTP
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:1167 [inlined]
 [2] macro expansion
   @ ./lock.jl:223 [inlined]
 [3] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1144
 [4] include(mod::Module, _path::String)
   @ Base ./Base.jl:419
 [5] include(x::String)
   @ HTTP /mnt/primal-db/primal-server/primal-caching-service/dev/HTTP.jl/src/HTTP.jl:1
 [6] top-level scope
   @ /mnt/primal-db/primal-server/primal-caching-service/dev/HTTP.jl/src/HTTP.jl:35
 [7] include
   @ ./Base.jl:419 [inlined]
 [8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
   @ Base ./loading.jl:1554
 [9] top-level scope
   @ stdin:1
in expression starting at /mnt/primal-db/primal-server/primal-caching-service/dev/HTTP.jl/src/Exceptions.jl:1
in expression starting at /mnt/primal-db/primal-server/primal-caching-service/dev/HTTP.jl/src/HTTP.jl:1
in expression starting at stdin:1
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types /nix/store/i92h4wy9ay6iays1dsg99qsjsjybyli1-julia-bin-1.8.5/share/julia/stdlib/v1.8/Pkg/src/Types.jl:67
 [2] precompile(ctx::Pkg.Types.Context, pkgs::Vector{String}; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API /nix/store/i92h4wy9ay6iays1dsg99qsjsjybyli1-julia-bin-1.8.5/share/julia/stdlib/v1.8/Pkg/src/API.jl:1435
 [3] precompile
   @ /nix/store/i92h4wy9ay6iays1dsg99qsjsjybyli1-julia-bin-1.8.5/share/julia/stdlib/v1.8/Pkg/src/API.jl:1072 [inlined]
 [4] #precompile#225
   @ /nix/store/i92h4wy9ay6iays1dsg99qsjsjybyli1-julia-bin-1.8.5/share/julia/stdlib/v1.8/Pkg/src/API.jl:1071 [inlined]
 [5] precompile (repeats 2 times)
   @ /nix/store/i92h4wy9ay6iays1dsg99qsjsjybyli1-julia-bin-1.8.5/share/julia/stdlib/v1.8/Pkg/src/API.jl:1071 [inlined]
 [6] top-level scope
   @ /mnt/primal-db/primal-server/primal-caching-service/pkg.jl:1
in expression starting at /mnt/primal-db/primal-server/primal-caching-service/pkg.jl:1

can we get a docker-compose.yml sample file?

I'd like to run this with docker-compose

Add filters to fetcher

Hello Team,

It'd be great to have an optional feature to fetch events only based on specified filters instead of fetching everything on nostr, it would enable the caching service to be hosted on smaller hosts, and only for usecase specific tasks, IMO it would make it very versatile.

Are there any plans to do this? I can help contribute to this area as well.

Running API Request examples

Hi, I was able to get it running and pulling data from the relays. How do I use the API? I see the examples but how do we make the request? Is there a Julia function I have to call on the REQ examples?

Also, can we send requests to our own instance endpoint? what would the url be?

Thanks!

Issues while building the primal-caching-service

building on a mac m1:
commit: 2abb1e7

when i run the command: nix --extra-experimental-features flakes --extra-experimental-features nix-command develop -c sh -c '$start_primal_caching_service'

i get:

ERROR: LoadError: The following 1 direct dependency failed to precompile:

HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3]

Error: Missing source file for HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types /nix/store/y1ka00ja531d4mi804kj0ggvcynixz49-julia-bin-1.9.4/share/julia/stdlib/v1.9/Pkg/src/Types.jl:69
 [2] precompile(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, timing::Bool, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
   @ Pkg.API /nix/store/y1ka00ja531d4mi804kj0ggvcynixz49-julia-bin-1.9.4/share/julia/stdlib/v1.9/Pkg/src/API.jl:1619
 [3] precompile(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API /nix/store/y1ka00ja531d4mi804kj0ggvcynixz49-julia-bin-1.9.4/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
 [4] precompile(pkgs::Vector{Pkg.Types.PackageSpec})
   @ Pkg.API /nix/store/y1ka00ja531d4mi804kj0ggvcynixz49-julia-bin-1.9.4/share/julia/stdlib/v1.9/Pkg/src/API.jl:145
 [5] precompile(; name::Nothing, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API /nix/store/y1ka00ja531d4mi804kj0ggvcynixz49-julia-bin-1.9.4/share/julia/stdlib/v1.9/Pkg/src/API.jl:171
 [6] precompile()
   @ Pkg.API /nix/store/y1ka00ja531d4mi804kj0ggvcynixz49-julia-bin-1.9.4/share/julia/stdlib/v1.9/Pkg/src/API.jl:162
 [7] top-level scope
   @ ~/Dev/test/primal-caching-service/pkg.jl:1
in expression starting at /Users/abhayraizada/Dev/test/primal-caching-service/pkg.jl:1

this is also the error i get inside the docker container when i run:

docker run -it --rm -v `pwd`:`pwd` -w `pwd` -p 8801:8801 -e PRIMALSERVER_HOST=0.0.0.0 nixos/nix nix --extra-experimental-features 'nix-command flakes' develop -c sh -c '$start_primal_caching_service'

can somebody help me with this?

Web client only connects to the cache

Primal only connects to its cache websocket 99% of the time, effectively making it read-only for me.

Liking, commenting, posting, zapping... nothing that involves writes works.

Just twice I saw it connect to a bunch of other relays which are not even my preferred relays, and those times likes etc did work (as expected).

Running it on Brave (Mac) but also tried Firefox - no luck.

getting error: ["NOTICE","6370032270415795","error"]

Getting this error when trying to send an event:

getting error: ["NOTICE","6370032270415795","error"]

┌ Error: handle_connection handler error│   exception =
│    HTTP.WebSockets.WebSocketError(HTTP.WebSockets.CloseFrameBody(1002, "Websocket handshake failed"))│    Stacktrace:│     [1] handshakeerror()│       @ HTTP.WebSockets /home/ubuntu/www/follows.lol/cache/dev/HTTP.jl/src/WebSockets.jl:323│     [2] upgrade(f::Main.CacheServer.var"#5#7", http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.ConnectionPool.Connection{Sockets.TCPSocket}}; suppress_close_error::Bool, maxframesize::Int64, maxfragmentation::Int64, kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})│       @ HTTP.WebSockets /home/ubuntu/www/follows.lol/cache/dev/HTTP.jl/src/WebSockets.jl:421│     [3] upgrade
│       @ /home/ubuntu/www/follows.lol/cache/dev/HTTP.jl/src/WebSockets.jl:419 [inlined]│     [4] (::HTTP.WebSockets.var"#11#12"{Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, Main.CacheServer.var"#5#7"})(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.ConnectionPool.Connection{Sockets.TCPSocket}})│       @ HTTP.WebSockets /home/ubuntu/www/follows.lol/cache/dev/HTTP.jl/src/WebSockets.jl:417
│     [5] #invokelatest#2│       @ ./essentials.jl:729 [inlined]│     [6] invokelatest│       @ ./essentials.jl:726 [inlined]│     [7] handle_connection(f::Function, c::HTTP.ConnectionPool.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing)│       @ HTTP.Servers /home/ubuntu/www/follows.lol/cache/dev/HTTP.jl/src/Servers.jl:447│     [8] (::HTTP.Servers.var"#16#17"{HTTP.WebSockets.var"#11#12"{Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, Main.CacheServer.var"#5#7"}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.ConnectionPool.Connection}, Int64, Nothing, Base.Semaphore, HTTP.ConnectionPool.Connection{Sockets.TCPSocket}})()│       @ HTTP.Servers ./task.jl:484└ @ HTTP.Servers /home/ubuntu/www/follows.lol/cache/dev/HTTP.jl/src/Servers.jl:461

Support regular Nostr filters

If Primal's caching service could support regular Nostr filters, it would greatly increase its value. It would appeal to self-hosters who would install this as part of self-hosted Nostr infrastructure. The current filters work only with Primal apps.

Missing a note in notification window

Hi,

Someone replied a note I posted but I don't see this reply in my notification window.
https://primal.net/e/note1ue8yl09nc7txvnxk6rf0jcl4lsyeyxlykqdhkd594sw9zvmey7wqg8pfjh

The reply from Rice user:

Issue while compiling

I'm having the following error is there any solution for it. I'm not familiar with nix.

~/primal-caching-service# nix --extra-experimental-features 'nix-command flakes' develop -c sh -c '$start_primal_caching_service'
make: *** No targets specified and no makefile found.  Stop.
ERROR: SystemError: opening file "/pkg.jl": No such file or directory
Stacktrace:
  [1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
    @ Base ./error.jl:176
  [2] #systemerror#80
    @ ./error.jl:175 [inlined]
  [3] systemerror
    @ ./error.jl:175 [inlined]
  [4] open(fname::String; lock::Bool, read::Nothing, write::Nothing, create::Nothing, truncate::Nothing, append::Nothing)
    @ Base ./iostream.jl:293
  [5] open
    @ ./iostream.jl:275 [inlined]
  [6] open(f::Base.var"#387#388"{String}, args::String; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Base ./io.jl:382
  [7] open
    @ ./io.jl:381 [inlined]
  [8] read
    @ ./io.jl:462 [inlined]
  [9] _include(mapexpr::Function, mod::Module, _path::String)
    @ Base ./loading.jl:1484
 [10] include
    @ ./Base.jl:419 [inlined]
 [11] exec_options(opts::Base.JLOptions)
    @ Base ./client.jl:283
 [12] _start()
    @ Base ./client.jl:522

Question: Is Primal the nostr client open source?

If so, where is the repo?

Also: Can I report an issue or are you not interested in user feedback?

Apologies for posting here. On nostr my messages are being ignored.

Fetching from DB kills the process and makes system unresponsive.

Hello Team,
Thank you so much for creating such a wonderful project.
It works well and fetches data correctly and dumps them into sqlite as expected but there are a few challenges I'm facing while trying to incorporate the system.

In julia shell, whenever I try to call event stats: event_stats(Main.DB.CacheStorage(directory="path_to_db_folder", Main.Nostr.EventId("event_id"), it renders the system irrresponsive and after a while, shell closes automatically.
Can someone help me how should I query the database in the var folder and how can i use the exposed functions correctly?

Open Feed issue

I am having some problem with Primal on Opera browser. I am not able to open the feed for ‘Latest, my follows’ nor ‘Latest, my Tribe’. I can open ‘Latest, my network’ and ‘Latest, global’. Any remedies to suggest?

What encoding is used for event_ids and author_pubkeys to store them as binary objects?

Hi Team,

I'm trying to fetch event ids but querying results in binary form data and I'm not sure how to decode them. Can someone tell me what encoding is used to convert the event_ids into BLOB format.

Also, what is the reason these can't be stored directly as string and are being stored as a blob instead?

`null` in requests are not handled properly

Normally when you have a null field in json it acts the same as if it were not included but the primal api will return an error

Error when compiling

I am getting this error when attempting to compile. I am guessing it is some linking error but not really familiar with anything in this stack.

$ nix develop -c sh -c '$start_primal_caching_service'
gcc -shared -g -fPIC -O2 -o libbech32.so -Ibech32/ref/c bech32.c
bech32.c:5:10: fatal error: segwit_addr.c: No such file or directory
    5 | #include "segwit_addr.c"
      |          ^~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:2: libbech32.so] Error 1
  0 dependencies successfully precompiled in 0 seconds. 83 already precompiled.
  1 dependency errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the package
  0 dependencies successfully precompiled in 0 seconds. 83 already precompiled.

ERROR: LoadError: The following 1 direct dependency failed to precompile:

HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3]

Error: Missing source file for HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types /nix/store/i92h4wy9ay6iays1dsg99qsjsjybyli1-julia-bin-1.8.5/share/julia/stdlib/v1.8/Pkg/src/Types.jl:67
 [2] precompile(ctx::Pkg.Types.Context, pkgs::Vector{String}; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API /nix/store/i92h4wy9ay6iays1dsg99qsjsjybyli1-julia-bin-1.8.5/share/julia/stdlib/v1.8/Pkg/src/API.jl:1435
 [3] precompile
   @ /nix/store/i92h4wy9ay6iays1dsg99qsjsjybyli1-julia-bin-1.8.5/share/julia/stdlib/v1.8/Pkg/src/API.jl:1072 [inlined]
 [4] #precompile#225
   @ /nix/store/i92h4wy9ay6iays1dsg99qsjsjybyli1-julia-bin-1.8.5/share/julia/stdlib/v1.8/Pkg/src/API.jl:1071 [inlined]
 [5] precompile (repeats 2 times)
   @ /nix/store/i92h4wy9ay6iays1dsg99qsjsjybyli1-julia-bin-1.8.5/share/julia/stdlib/v1.8/Pkg/src/API.jl:1071 [inlined]
 [6] top-level scope
   @ ~/primal-caching-service/pkg.jl:1
in expression starting at /root/primal-caching-service/pkg.jl:1

Not working on chrome with Alby

I tried to explore primal.net on Chrome using Alby, but even after completing the actions of getPublicKey and the SignEvent, I'm not able to see any effect like I'm not able to post, follow, react, etc.
I have recorded it's working , please find it here.

Trouble building/running the caching service with nix

Hey guys, I'm new to nix and thus unsure how to proceed here. I've installed nix on Ubuntu 22.04.2 LTS and cloned the repo. Then from the repo directory I'm running nix --extra-experimental-features 'nix-command flakes' develop -c sh -c '$start_primal_caching_service'

On the first run, it installed most of the dependencies but errored on installing HTTP. Following runs of the same command return the following:

gcc -shared -g -fPIC -O2 -o libbech32.so -Ibech32/ref/c bech32.c
bech32.c:5:10: fatal error: segwit_addr.c: No such file or directory
    5 | #include "segwit_addr.c"
      |          ^~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:2: libbech32.so] Error 1
  0 dependencies successfully precompiled in 0 seconds. 83 already precompiled.
  1 dependency errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the package
  0 dependencies successfully precompiled in 0 seconds. 83 already precompiled.

ERROR: LoadError: The following 1 direct dependency failed to precompile:

HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3]

Error: Missing source file for HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types /nix/store/i92h4wy9ay6iays1dsg99qsjsjybyli1-julia-bin-1.8.5/share/julia/stdlib/v1.8/Pkg/src/Types.jl:67
 [2] precompile(ctx::Pkg.Types.Context, pkgs::Vector{String}; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API /nix/store/i92h4wy9ay6iays1dsg99qsjsjybyli1-julia-bin-1.8.5/share/julia/stdlib/v1.8/Pkg/src/API.jl:1435
 [3] precompile
   @ /nix/store/i92h4wy9ay6iays1dsg99qsjsjybyli1-julia-bin-1.8.5/share/julia/stdlib/v1.8/Pkg/src/API.jl:1072 [inlined]
 [4] #precompile#225
   @ /nix/store/i92h4wy9ay6iays1dsg99qsjsjybyli1-julia-bin-1.8.5/share/julia/stdlib/v1.8/Pkg/src/API.jl:1071 [inlined]
 [5] precompile (repeats 2 times)
   @ /nix/store/i92h4wy9ay6iays1dsg99qsjsjybyli1-julia-bin-1.8.5/share/julia/stdlib/v1.8/Pkg/src/API.jl:1071 [inlined]
 [6] top-level scope
   @ ~/primal-caching-service/pkg.jl:1
in expression starting at /home/jg/primal-caching-service/pkg.jl:1

Question: Does this store events in SQLite?

I'm trying to understand this code. I see SQLite being used. Is that really the main data store? You're indexing almost the entirety of Nostr with SQLite? If so, how?