Ursa, a decentralized content delivery network that is censorship-resistant and trustless without compromise on throughput and latency
Home Page: https://fleek.network
License: Apache License 2.0
The rpc get should have query params and the requested cid should be urlencoded
add subcommands to cli for:
also, stop exposing endpoints that should be accessible through CLI
file upload via http is not supported by json-rpc. Add Http server to current rpc-server, both running on the same port.
For a given content id that is not stored, and requested via rpc, we don't handle the error correctly
CLI Log:
oz@instance-4:~$ ./ursa-bin rpc get bafybeifyjj2bjhtxmp235vlfeeiy7sz6rzyx3lervfk3ap2nyn4rggqgei asdf
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: NoResponse: server did not send a response', /home/oz/dev/fleek/ursa/crates/ursa-rpc-client/src/lib.rs:69:22
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Node Log:
...
2022-12-09T01:14:59.201874Z ERROR ursa_network::behaviour: Failed to retrieve block bafybeifyjj2bjhtxmp235vlfeeiy7sz6rzyx3lervfk3ap2nyn4rggqgei.
2022-12-09T01:14:59.202010Z ERROR ursa_network::service: [BehaviourEvent::Bitswap] - block not found.
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: The bitswap failed, please check server logs The requested block with cid Cid(bafybeifyjj2bjhtxmp235vlfeeiy7sz6rzyx3lervfk3ap2nyn4rggqgei) is not found with any peers', /home/oz/dev/fleek/ursa/crates/ursa-rpc-server/src/api.rs:157:49
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
...
Currently, the advertisement chain is stored in memory
Either come up with a mechanism to restore all the ads when a node is restarted with new id
Or use persistent storage for advertisement chain - trade off here memory vs storage
Nodes should perform a kademlia query every few minutes to discover new peers
The rocksdb crate only seems to support vendoring zstd/zlib/etc, which means if a Rust crate depends on rocksdb and also pulls in zstd/zlib/etc, it'll end up with symbol conflicts between the rocksdb vendored zstd/zlib/etc and the other version they include.
This happens only when building in debug mode and will be solved once we upgrade our dependencies to version >=0.19.
rust-rocksdb/rust-rocksdb#514
rust-rocksdb/rust-rocksdb#535.
fvm_ipld_car doesn't handle linking between roots. This is important when storing car filesUse https://github.com/seanmonstar/warp
if the block is not found and the peer list for bitswap is empty, the thread panics and oneshot channel is not closed.
handle this case and complete the channel send to be able to close it properly
NodeNetworkInterface to remove functions that aren't api functions e.g get_dataUse Prometheus to collect node metrics
https://github.com/prometheus/client_rust
https://crates.io/crates/libp2p-metrics
-- # number of peers
-- # of cids we are caching
-- todo
resolve content from cache node
handle the case where the requested block is not found from any peer
Rust currently doesn't allow a direct way of overriding major versions of transitive dependencies.
One workaround was to fork dependency repos and override those versions in Cargo.toml
the related issue on rust repo: rust-lang/cargo#5640
if and once this is resolved, we should remove forked repos and use overriding technique.
The current forked repos are(please add to list if we add more):
Network tests need updating as they are conflicting and immediately failing. Attempted running tests on current main as well as #70
attempting to run tests with:
$ cargo test -r --allgives:
Finished release [optimized] target(s) in 0.25s
Running unittests src/main.rs (target/release/deps/ursa-5979bd09da2b25a1)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Running unittests src/main.rs (target/release/deps/ursa_gateway-4a2da74aaf6d4012)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Running unittests src/lib.rs (target/release/deps/ursa_index_provider-f4def0148787e8db)
running 3 tests
test signed_head::tests::test_decode_signed_msg ... ok
test signed_head::tests::test_sign_head ... ok
test provider::tests::test_create_ad ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.02s
Running unittests src/lib.rs (target/release/deps/ursa_metrics-7057e3ef20b689e6)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Running unittests src/lib.rs (target/release/deps/ursa_network-d6c778180acd4664)
running 14 tests
test service::tests::get_block_local ... FAILED
test codec::protocol::tests::test_read_response ... FAILED
test codec::protocol::tests::test_write_request ... FAILED
test codec::protocol::tests::test_write_response ... FAILED
test codec::protocol::tests::test_read_request ... FAILED
test service::tests::add_block ... FAILED
test service::tests::test_bitswap_get ... FAILED
test service::tests::test_bitswap_sync ... FAILED
test service::tests::test_bitswap_get_block_not_found ... FAILED
test service::tests::test_network_mdns ... FAILED
test service::tests::test_network_gossip ... FAILED
test service::tests::test_network_discovery ... FAILED
test service::tests::test_network_req_res ... FAILED
test service::tests::test_network_start ... ok
failures:
---- service::tests::get_block_local stdout ----
thread 'service::tests::get_block_local' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14
---- codec::protocol::tests::test_read_response stdout ----
thread 'codec::protocol::tests::test_read_response' panicked at 'not yet implemented', crates/ursa-network/src/codec/protocol.rs:145:9
---- codec::protocol::tests::test_write_request stdout ----
thread 'codec::protocol::tests::test_write_request' panicked at 'not yet implemented', crates/ursa-network/src/codec/protocol.rs:150:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- codec::protocol::tests::test_write_response stdout ----
thread 'codec::protocol::tests::test_write_response' panicked at 'not yet implemented', crates/ursa-network/src/codec/protocol.rs:155:9
---- codec::protocol::tests::test_read_request stdout ----
thread 'codec::protocol::tests::test_read_request' panicked at 'not yet implemented', crates/ursa-network/src/codec/protocol.rs:140:9
---- service::tests::add_block stdout ----
thread 'service::tests::add_block' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14
---- service::tests::test_bitswap_get stdout ----
thread 'service::tests::test_bitswap_get' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14
---- service::tests::test_bitswap_sync stdout ----
thread 'service::tests::test_bitswap_sync' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14
---- service::tests::test_bitswap_get_block_not_found stdout ----
thread 'service::tests::test_bitswap_get_block_not_found' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14
---- service::tests::test_network_mdns stdout ----
thread 'service::tests::test_network_mdns' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14
---- service::tests::test_network_gossip stdout ----
thread 'service::tests::test_network_gossip' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14
---- service::tests::test_network_discovery stdout ----
thread 'service::tests::test_network_discovery' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14
---- service::tests::test_network_req_res stdout ----
thread 'service::tests::test_network_req_res' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14
failures:
codec::protocol::tests::test_read_request
codec::protocol::tests::test_read_response
codec::protocol::tests::test_write_request
codec::protocol::tests::test_write_response
service::tests::add_block
service::tests::get_block_local
service::tests::test_bitswap_get
service::tests::test_bitswap_get_block_not_found
service::tests::test_bitswap_sync
service::tests::test_network_discovery
service::tests::test_network_gossip
service::tests::test_network_mdns
service::tests::test_network_req_res
test result: FAILED. 1 passed; 13 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
error: test failed, to rerun pass `-p ursa-network --lib`
We should probably have the tests run automatically on every pull request to avoid failing tests on our main branch in the future. Alongside this, it would be nice to have a runner status badge on the readme document to show if the latest is successfully passing at a quick glance
👋 fleek-folk! Congrats on your recent raise & this new project!
I work on iroh a rust implementation of IPFS (github: https://github.com/n0-computer/iroh), and looking through your codebase it might be fun to share notes, and maybe even a little bit of code 😄.
Some overlap I'm already seeing:
#79
n0-computer/iroh#476
If you have any interest in chatting, I'd love to set up a call to see if there are places where we might be able to help each other out. With that said, I totally get it if y'all just need to move fast & keep things in your own codebase. If that's the case, we can always share notes!
client setup for rpc-server.
yeah nice angle i like it. Only one note, we shouldn't fork that one. I guess yall mean follow the structure <- which sounds better. Because we can just have a bunch of tests there and with a testground locally can pull these tests and run them. So decoupled from our code (meaning we can import using github into the crates).
Another note here are some tests from a few months ago that should be covered:
The test plans will tackle each of the node's different behaviors.
These behaviors include:
investigate upgrading libp2p to 0.46.1 is breaking bitswap implementations.
With content put through rpc to node A, and get through rpc cli through a node B which doesn't have the content and bitswap performs a query, the cli never resolves and hangs indefinitely.
Running this command from node B:
./ursa-bin rpc get bafybeifyjj2bjhtxmp235vlfeeiy7sz6rzyx3lervfk3ap2nyn4rggqgei asdf
node B logs (with a debug placed on all bitswap events):
2022-12-08T04:43:26.316279Z INFO ursa_rpc_server::api: getting and storing the file at: asdf
BitswapEvent: Progress(QueryId(0), 1)
BitswapEvent: Complete(QueryId(0), Ok(()))
and the rpc command hangs indefinitely from there
Cancelling and doing a second get request resolves extremely quickly and works as expected, so the node is getting the content but just not responding to rpc on that first request
Prioritize Typescript SDK
###Initial requirements
Async? Should we be runtime agnostic
build docker image after closing PR on main
push docker image to github registry
push to gh registry
run rust tests
When dialling to bootstrap nodes for mainnet indexer, there are some protocol differences. This prevent us from connecting to global indexers for gossiping the indexing announcements.
Investigate and fix this issue.
Also test announcing to indexers via gossipsub
When using the config provided in the README, the paths specified ~/... do not get parsed correctly and result in a folder created literally called ~. Should parse this correctly as the HOME dir
For now, requesting a node over https returns an error related with certs
Error: SSL peer certificate or SSH remote key was not OK
The ursa bin has a build.rs that populates the default config into ~/.ursa/. This should be done automatically when the node runs if the config doesn't exist, so it's redundant.
Other reasoning:
a basic first draft of the rpc server
cahce peers behind Nat dialing bootstrap are not discovered by the other peers.
Related to this issue raised in libp2p-bitswap repo
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.