Comments (2)
Seems reasonable. The main tradeoff here is that we might over-read data from disk which may/may not be that important.
The reprovider strategies are here
Lines 130 to 139 in e22f47a
Probably the easiest way to do this is:
- "roots" -> keep doing the same thing
- "pinned" -> do a first pass using the "roots" function and then do this function (maybe keep something like a bloom filter or map so we don't readvertise the same data twice in exchange for some more memory)
- If the plan is to use a map here it'd probably be better to rewrite the function to enumerate the roots and then put them into a queue so we don't have to hit the datastore twice
- "all" -> do a first pass using the "roots" function, then do the blockstore enumeration
- Could do roots -> pinned -> all but if most of the data is pinned it'll require enumerating the blockstore twice which could be quite expensive. This way the worst case is if the blockstore is lots of tiny objects in which case the datastore and blockstore are both enumerated (might still be annoying, but likely less bad)
from kubo.
From conversation with @lidel:
Something we could also do here is doing a local walk (i.e. don't do any remote fetching) of MFS and announcing the file/folder roots first (along with the other "roots").
Tracked in #10386
from kubo.
Related Issues (20)
- Move and deprecate `bitswap reprovide` to `routing reprovide`
- CVE-2024-22189 quic-go: memory exhaustion attack HOT 1
- Lower healthcheck interval
- Release 0.29 HOT 6
- How do I update the public IP or the config file programmatically HOT 1
- Creating node in Android with gomobile & kubo is failing by saying `GETENT` is not found in path
- bug: kubo doesn't announce itself as provider HOT 4
- docs: add GOLOG_LOG_LEVEL debug logging info
- why i upload jpg file, tell me error like this invalid character 'ΓΏ' looking for beginning of value HOT 1
- i clone kubo source code when i run kubo ,tell this error:cannot acquire lock: creation of lock D:\workdir\kubo\127.0.0.1:5001\repo.lock failed: The filename, directory name, or volume label syntax is incorrect
- IPFS cat unable to retrieve updated information unless daemon is restarted. HOT 1
- Get key by name
- websocket: failed to close network connection: close tcp HOT 5
- Describe environment variable names/expected values in config docs HOT 1
- Cannot retrieve content by addressing with SHA256 (raw binary?) HOT 1
- api/v0/resolve returns invalid JSON
- Release 0.30
- IPFS RPC API documentation HOT 1
- Ipfs kubo node memory usage increases endlessly HOT 1
- /api/v0/get can't save file and not equal cmd: ipfs get
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubo.