Comments (12)
I mean... no. this would be very difficult (and probably not desirable) to do exactly since pickle is very Python specific. However, doing something similar, where values are broken up would be relatively straight forward (i've done it for Dalli). Not sure if this makes sense as a separate library/code snippet. If you get around to this I'll definitely appreciate a pull request. Otherwise, I might get around to it.
In general, there are some tricky tradeoffs with the kinds of strategies you might use to spread values across multiple keys, that I'm not sure how best to generalize (as opposed to giving code snippets for how to write a custom thing for each app, or a higher level library with multiple options or something). But very interested in seeing what can be done.
from memjs.
Can you post the code you used? The implementation doesnt matter (pickle vs something else) as long i can replicate the result.
Sent from my iPhone
On Sep 21, 2013, at 9:29 PM, Amit Levy [email protected] wrote:
I mean... no. this would be very difficult (and probably not desirable) to do exactly since pickle is very Python specific. However, doing something similar, where values are broken up would be relatively straight forward (i've done it for Dalli). Not sure if this makes sense as a separate library/code snippet. If you get around to this I'll definitely appreciate a pull request. Otherwise, I might get around to it.
In general, there are some tricky tradeoffs with the kinds of strategies you might use to spread values across multiple keys, that I'm not sure how best to generalize (as opposed to giving code snippets for how to write a custom thing for each app, or a higher level library with multiple options or something). But very interested in seeing what can be done.
—
Reply to this email directly or view it on GitHub.
from memjs.
Bump
from memjs.
Just curious why you closed this since the issue isn't solved.
from memjs.
Hrm... mostly I was just doing cleaning, and hadn't touched this issue in almost a year... but..
I wasn't convinced this is a good feature to have in this level of library since it would need to provide a specific choice amongst a set of imperfect semantics: what happens when one of the keys is evicted? do you store keys sequntially (i.e. mykey_1
, mykey_2
, etc) and read until you get a miss or have a base key that contains a list of subkeys? do you split up values automatically if it's too large to fit or do you have a separate command?
My hunch is that all of the answers are correct for some people and I'm not sure there is a "good" one for most.
However, I can mention what I've done in a certain app algorithmically:
set:
- split data into chunks < 1MB
- set base key to the number of chunks, retrieving the version of the base key (this is not supported in
memjs
, but could and should be) - Do a compare-and-swap with the version number from the base key on each of the subsequent keys. If it fails with an earlier version number, redo using the new version number. If it fails with a later version number, abort -- another thread has override your previous work already.
get:
- Get the base key with version number
- Get all referenced subkeys 0-n ensuring until the version doesn't match. If the non-matched version is higher, start over as someone has overriden the value.
from memjs.
Well, I'm far from an expert on this topic, but why not just do it how the python version does it? There has to be a nodejs equivalent of pickle, right? At the least, can you post code showing how you worked around this issue before?
from memjs.
And to answer one of the questions, you would select a spread
mode upon instantiating memjs
. In that mode, any values > max size would be split up automatically. Also, what's wrong with reading until you get a miss? Couldn't you just put it in a loop and place that in a try/except block?
from memjs.
pickle doesn't help much here, it's just a serialization format, and in any case, memjs only deals with values of type String
or Buffer
which are already serialized. So as far as splitting things up, that's just:
// largeValue is a `Buffer` to split, maxSize is the maximum size of any part
function split(largeValue, maxSize) {
var parts = []
while(largeValue.length > 0) {
parts.push(largeValue.slice(0, maxSize);
largeValue = largeValue.slice(maxSize);
}
return parts;
}
Here is a gist of some ruby skeleton code for spreading this across the cache like this with Dalli
: https://gist.github.com/alevy/5512907
from memjs.
The problem with just iterating until you get a miss is two-fold:
- When you actually get a miss, how do you know if the miss is because you finished reading all the parts? Or because that sub-key was evicted in the cache? So you need some sort of terminator sequence, but then what if that sequence happens to exist in the value itself, so you need to escape it in the value, etc...
- It's very slow to retrieve a long value -- you have to wait for a response from each subkey in order to proceed to the next one (you could mitigate this by batching, say, 5 at a time, but that sucks for values smaller than 5 sub-keys). Ideally, you'd like to be able to run most of the requests "quietly" such that your not waiting a round trip on each request until sending the next one. This is the difference between retrieving a large value in <10ms vs. many 100's of ms.
I think spreading values automatically is problematic because it breaks performance assumptions. I expect that when I call a memcache function on a low-level library, I get the performance of a single request (should be on the order 1ms). If all of a sudden I get a 2x or more performance hit because my value is slightly larger, that's less intuitive behavior (for a performance sensitive library) IMO than an error.
from memjs.
Having said all that, I think this discussion you've helped highlight some missing lower level features that would make doing this easier and would also be useful on their own (specifically having a way to get at the version of get and set response, as well as supporting compare-and-swap). I'll open new issues for those (and /cc you). I don't mind at all including a utility function (although probably in a different namespace) to split up values in a particular way -- we've basically already pseudo-coded the algorithm in this thread :).
from memjs.
I think you've misunderstood me. Spreading values wouldn't happen automatically. It would be an option in the setup, e.g., client = memjs.Client.create({spread: True})
. By default, spread would be False.
from memjs.
Oh interesting. OK, i don't mind that as much... reopening
from memjs.
Related Issues (20)
- timeout but failed to retry until restarting program HOT 28
- Is it possible to use from the browser? HOT 1
- Operation timeouts do not work as expected HOT 3
- How to clear memcached server from nodejs? HOT 2
- Denial of Service vulnerability HOT 1
- memJS terminates connection to coherence HOT 1
- Feature: Key wrapper
- Get without callback returns incorrect signature HOT 2
- Possibility to get all errors from server
- Do we have option to use limit and offset HOT 1
- Application exits if memcached server restarts HOT 1
- Can we delete many keys? HOT 1
- unix socket support HOT 1
- MCRouter issue HOT 3
- Add node autodiscovery HOT 1
- TLS support for memjs
- Sporadic bursts of error: socket timed out waiting on response
- Trying an operation after calling `.quit()` doesn't trigger any errors HOT 3
- Possible bug in signature check HOT 2
- Possible support for ASCII protocol? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from memjs.