artemgr / rados.hi.rs Goto Github PK
View Code? Open in Web Editor NEWOpinionated high-level wrapper for librados (Ceph).
License: MIT License
Opinionated high-level wrapper for librados (Ceph).
License: MIT License
When handling two-level hierarchies, it's often useful to limit parallelism.
Here's what I often do, in pseudocode:
level1 = ids.map (|id| ctx.read (id)) .collect() // Load the first level in parallel.
for (idx in level1) {
level2 = idx.map (|id| ctx.read (id)) .collect() // Load the second level in parallel.
...
}
Now, I don't need the entirety of the first level loaded when I'm handling the second level. In fact, loading the first level might be seen as inefficient in terms of memory use and ordering of the AIO operations. What I need is prefetching some (maybe just a single entry) of the first level in parallel while focusing on the second level.
I wonder what the proper/best API for this might look like.
P.S. There seems to be some explicit parallelism adaptors already. On one hand, join_all, which runs all futures in parallel, on another hand, buffer_unordered, which runs a fixed chunk of the futures in parallel. We need something in between, an adapter that eventually runs all the futures, but not more than a couple of them at a time.
P.S. Another use case, in pseudocode:
let oids = [...];
let stats = parallelize (99, oid.map (ctx.stat (oid)));
for bytes in parallelize (9, stats.map (ctx.read (oid, size))) {...}
We often have a collection that we want to generate futures from, but we don't want to generate all the futures immediately (that makes the app pause and eats RAM), so there needs to be some kind of a buffer that applies a map
in advance but only for a limited number of entries. This seems to be entirely generic, e.g. we take a simple Iterator
and produce a different Iterator
that buffers a fixed number of entries, that's all.
This is great work! I've been working on a very similar aio setup using libffi. I've been stuck trying to figure out how to unpark the task. Would you consider PRing your work back into ceph-rust?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.