fleek-network / ursa Goto Github PK

Ursa, a decentralized content delivery network that is censorship-resistant and trustless without compromise on throughput and latency

Home Page: https://fleek.network

License: Apache License 2.0

Rust 87.76% Makefile 0.52% Shell 0.81% Dockerfile 0.76% HCL 6.71% JavaScript 0.46% TypeScript 0.83% Solidity 2.16%

cdn ipfs ipld narwhal rust libp2p bullshark

ursa's People

Contributors

Stargazers

Watchers

ursa's Issues

chore: persistent storage for advertisement chains

Currently, the advertisement chain is stored in memory
Either come up with a mechanism to restore all the ads when a node is restarted with new id
Or use persistent storage for advertisement chain - trade off here memory vs storage

bug: inproper error handling for bitswap content id not found

For a given content id that is not stored, and requested via rpc, we don't handle the error correctly

CLI Log:

oz@instance-4:~$ ./ursa-bin rpc get bafybeifyjj2bjhtxmp235vlfeeiy7sz6rzyx3lervfk3ap2nyn4rggqgei asdf
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: NoResponse: server did not send a response', /home/oz/dev/fleek/ursa/crates/ursa-rpc-client/src/lib.rs:69:22
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Node Log:

...
2022-12-09T01:14:59.201874Z ERROR ursa_network::behaviour: Failed to retrieve block bafybeifyjj2bjhtxmp235vlfeeiy7sz6rzyx3lervfk3ap2nyn4rggqgei.
2022-12-09T01:14:59.202010Z ERROR ursa_network::service: [BehaviourEvent::Bitswap] - block not found.
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: The bitswap failed, please check server logs The requested block with cid Cid(bafybeifyjj2bjhtxmp235vlfeeiy7sz6rzyx3lervfk3ap2nyn4rggqgei) is not found with any peers', /home/oz/dev/fleek/ursa/crates/ursa-rpc-server/src/api.rs:157:49
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
...

chore: query global indexer

feat: dag sync

full dag syncing for car files
Add sync query for bitswap

chore: add query params to rpc get

The rpc get should have query params and the requested cid should be urlencoded

bug: response from rpc server

error code from get query
return cid from the put file and put car

bug: improper path parsing for `~`

When using the config provided in the README, the paths specified ~/... do not get parsed correctly and result in a folder created literally called ~. Should parse this correctly as the HOME dir

feat: gateway content cache

chore: infra orchestration

chore: github workflow

build docker image after closing PR on main
push docker image to github registry
push to gh registry
run rust tests

network indexing

chore: update network behaviours - libp2p

feat: global metrics

chore: improve blockstore cache, introduce LRU/LFU cache

currently fvm_ipld_car doesn't handle linking between roots. This is important when storing car files
most likely LRU

chore: bitswap and libp2p new version

investigate upgrading libp2p to 0.46.1 is breaking bitswap implementations.

feat: add subcommands to cli

add subcommands to cli for:

put file on the network

also, stop exposing endpoints that should be accessible through CLI

feat: rpc-server

a basic first draft of the rpc server

bug: handle thread panic when peer list is empty for bitswap

if the block is not found and the peer list for bitswap is empty, the thread panics and oneshot channel is not closed.
handle this case and complete the channel send to be able to close it properly

improvement: unnecessary build.rs

The ursa bin has a build.rs that populates the default config into ~/.ursa/. This should be done automatically when the node runs if the config doesn't exist, so it's redundant.

Other reasoning:

the build.rs adds ~500 crates to compile for a total of 1300+ crates on a fresh clean build
+30s build time on a cached build, just for it to create a folder and a file

A simple proxy/gateway

Use https://github.com/seanmonstar/warp

create a simple proxy that uses the rpc-client to interact with the edge node
deploy it using fly.io
might just use nginx for the time being

https://www.digitalocean.com/community/tools/nginx

feat: ursa SDK client (TS/Rust)

Typescript
Rust

Prioritize Typescript SDK

###Initial requirements

define json rpc api

basic rpc functionality

Open Topic

Async? Should we be runtime agnostic

feat: prometheus setup

chore: automatic kademlia walks

Nodes should perform a kademlia query every few minutes to discover new peers

possible collab & code sharing with n0-computer/iroh?

👋 fleek-folk! Congrats on your recent raise & this new project!

I work on iroh a rust implementation of IPFS (github: https://github.com/n0-computer/iroh), and looking through your codebase it might be fun to share notes, and maybe even a little bit of code 😄.

Some overlap I'm already seeing:
#79
n0-computer/iroh#476

If you have any interest in chatting, I'd love to set up a call to see if there are places where we might be able to help each other out. With that said, I totally get it if y'all just need to move fast & keep things in your own codebase. If that's the case, we can always share notes!

improve: rewrite network tests

chore: protocol issues with dialling bootstrap nodes for mainnet indexer

When dialling to bootstrap nodes for mainnet indexer, there are some protocol differences. This prevent us from connecting to global indexers for gossiping the indexing announcements.
Investigate and fix this issue.
Also test announcing to indexers via gossipsub

feat: multiplex server(add http to current rpc server)

file upload via http is not supported by json-rpc. Add Http server to current rpc-server, both running on the same port.

feat: gateway index workers

Motivation:

#121

How:

jobs fetching results from indexers and propagating results to caches.

PR:

#195

bug: issues with peer discovery

cahce peers behind Nat dialing bootstrap are not discovered by the other peers.

chore: stream improvements

recursive dag traversal
make stream handler to be non blocking
send the first byte as soon as the chunk of data is ready
refactor NodeNetworkInterface to remove functions that aren't api functions e.g get_data

improve: index provider

feat: gateway-nodes infra

proxy: Set HTTPS properly

For now, requesting a node over https returns an error related with certs

Error: SSL peer certificate or SSH remote key was not OK

chore: handle "block not found" for bitswap

handle the case where the requested block is not found from any peer

feat: file streaming via rpc

network tests and pr runner

Network tests need updating as they are conflicting and immediately failing. Attempted running tests on current main as well as #70

attempting to run tests with:

$ cargo test -r --all

gives:

errors from SimpleLogger that the init has already been run
commenting out the simple logger, we get errors from rocks db a lock is already established on the database

Finished release [optimized] target(s) in 0.25s
     Running unittests src/main.rs (target/release/deps/ursa-5979bd09da2b25a1)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/main.rs (target/release/deps/ursa_gateway-4a2da74aaf6d4012)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/release/deps/ursa_index_provider-f4def0148787e8db)

running 3 tests
test signed_head::tests::test_decode_signed_msg ... ok
test signed_head::tests::test_sign_head ... ok
test provider::tests::test_create_ad ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.02s

     Running unittests src/lib.rs (target/release/deps/ursa_metrics-7057e3ef20b689e6)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/release/deps/ursa_network-d6c778180acd4664)

running 14 tests
test service::tests::get_block_local ... FAILED
test codec::protocol::tests::test_read_response ... FAILED
test codec::protocol::tests::test_write_request ... FAILED
test codec::protocol::tests::test_write_response ... FAILED
test codec::protocol::tests::test_read_request ... FAILED
test service::tests::add_block ... FAILED
test service::tests::test_bitswap_get ... FAILED
test service::tests::test_bitswap_sync ... FAILED
test service::tests::test_bitswap_get_block_not_found ... FAILED
test service::tests::test_network_mdns ... FAILED
test service::tests::test_network_gossip ... FAILED
test service::tests::test_network_discovery ... FAILED
test service::tests::test_network_req_res ... FAILED
test service::tests::test_network_start ... ok

failures:

---- service::tests::get_block_local stdout ----
thread 'service::tests::get_block_local' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14

---- codec::protocol::tests::test_read_response stdout ----
thread 'codec::protocol::tests::test_read_response' panicked at 'not yet implemented', crates/ursa-network/src/codec/protocol.rs:145:9

---- codec::protocol::tests::test_write_request stdout ----
thread 'codec::protocol::tests::test_write_request' panicked at 'not yet implemented', crates/ursa-network/src/codec/protocol.rs:150:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- codec::protocol::tests::test_write_response stdout ----
thread 'codec::protocol::tests::test_write_response' panicked at 'not yet implemented', crates/ursa-network/src/codec/protocol.rs:155:9

---- codec::protocol::tests::test_read_request stdout ----
thread 'codec::protocol::tests::test_read_request' panicked at 'not yet implemented', crates/ursa-network/src/codec/protocol.rs:140:9

---- service::tests::add_block stdout ----
thread 'service::tests::add_block' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14

---- service::tests::test_bitswap_get stdout ----
thread 'service::tests::test_bitswap_get' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14

---- service::tests::test_bitswap_sync stdout ----
thread 'service::tests::test_bitswap_sync' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14

---- service::tests::test_bitswap_get_block_not_found stdout ----
thread 'service::tests::test_bitswap_get_block_not_found' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14

---- service::tests::test_network_mdns stdout ----
thread 'service::tests::test_network_mdns' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14

---- service::tests::test_network_gossip stdout ----
thread 'service::tests::test_network_gossip' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14

---- service::tests::test_network_discovery stdout ----
thread 'service::tests::test_network_discovery' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14

---- service::tests::test_network_req_res stdout ----
thread 'service::tests::test_network_req_res' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', crates/ursa-network/src/service.rs:494:14


failures:
    codec::protocol::tests::test_read_request
    codec::protocol::tests::test_read_response
    codec::protocol::tests::test_write_request
    codec::protocol::tests::test_write_response
    service::tests::add_block
    service::tests::get_block_local
    service::tests::test_bitswap_get
    service::tests::test_bitswap_get_block_not_found
    service::tests::test_bitswap_sync
    service::tests::test_network_discovery
    service::tests::test_network_gossip
    service::tests::test_network_mdns
    service::tests::test_network_req_res

test result: FAILED. 1 passed; 13 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s

error: test failed, to rerun pass `-p ursa-network --lib`

We should probably have the tests run automatically on every pull request to avoid failing tests on our main branch in the future. Alongside this, it would be nice to have a runner status badge on the readme document to show if the latest is successfully passing at a quick glance

chore: dockerize the cli/node

dockerize cli
cache build on repeated builds!
fix running cli with correct params
some misc items

chore: bitswap consistency issue

Related to this issue raised in libp2p-bitswap repo

ipfs-rust/libp2p-bitswap#29

chore: remove forked dependencies

Rust currently doesn't allow a direct way of overriding major versions of transitive dependencies.
One workaround was to fork dependency repos and override those versions in Cargo.toml

the related issue on rust repo: rust-lang/cargo#5640

if and once this is resolved, we should remove forked repos and use overriding technique.

The current forked repos are(please add to list if we add more):

feat: node metrics

Use Prometheus to collect node metrics

https://github.com/prometheus/client_rust

https://crates.io/crates/libp2p-metrics

-- # number of peers
-- # of cids we are caching
-- todo

feat: gateway handle signal/gracefully shutdown

feat: rpc-client

client setup for rpc-server.

feat: gateway resolver

resolve content from cache node

bug: ursa cli hangs requesting content not stored on the node

With content put through rpc to node A, and get through rpc cli through a node B which doesn't have the content and bitswap performs a query, the cli never resolves and hangs indefinitely.

Running this command from node B:

./ursa-bin rpc get bafybeifyjj2bjhtxmp235vlfeeiy7sz6rzyx3lervfk3ap2nyn4rggqgei asdf

node B logs (with a debug placed on all bitswap events):

2022-12-08T04:43:26.316279Z  INFO ursa_rpc_server::api: getting and storing the file at: asdf
BitswapEvent: Progress(QueryId(0), 1)
BitswapEvent: Complete(QueryId(0), Ok(()))

and the rpc command hangs indefinitely from there

Cancelling and doing a second get request resolves extremely quickly and works as expected, so the node is getting the content but just not responding to rpc on that first request

feat: integrate StoreTheIndex

Linking error due to multiple versions of zstd library

The rocksdb crate only seems to support vendoring zstd/zlib/etc, which means if a Rust crate depends on rocksdb and also pulls in zstd/zlib/etc, it'll end up with symbol conflicts between the rocksdb vendored zstd/zlib/etc and the other version they include.

This happens only when building in debug mode and will be solved once we upgrade our dependencies to version >=0.19.

rust-rocksdb/rust-rocksdb#514
rust-rocksdb/rust-rocksdb#535.

test: Create testplans for the cache nodes

yeah nice angle i like it. Only one note, we shouldn't fork that one. I guess yall mean follow the structure <- which sounds better. Because we can just have a bunch of tests there and with a testground locally can pull these tests and run them. So decoupled from our code (meaning we can import using github into the crates).

Another note here are some tests from a few months ago that should be covered:

peers
routing
replication
traffic

The test plans will tackle each of the node's different behaviors.

These behaviors include:

Gossip
Ping
Identify
Bitswap
DiscoveryBehaviour
RequestResponse (rpc) how we handle commands to our network

fleek-network / ursa Goto Github PK

ursa's People

Contributors

Stargazers

Watchers

Forkers

ursa's Issues

basic rpc functionality

Open Topic

Recommend Projects

Recommend Topics

Recommend Org