Comments (4)
Every value over the size of N will be split into smaller chunks of size N (last chunk might be less than N). The original key becomes the 'index' key which stores a random suffix and total number of chunks; the chunks are stored at modified keys that includes the same random suffix and the chunk id.
The random suffix is needed for consistency - you can simply remove the original key and the chunks will be 'deleted' - there's no way to access them without knowing the random suffix. It also takes care of simultaneous sets, since only one key will win the race, and only its random suffix will be valid.
The way we deployed it on a live system was in stages. First we deployed reads only - if you set N to some large value (like 1000000000), the logic is still enabled on the read path, but will not actually split any values. This makes sure that all clients can understand split values once we start writing them.
Second stage was lowering N to actually start splitting the values. The exact value we use is 524288.
Note that all chunks will be sent to the same memcache box as the original key would be, so that means you're still transferring the same amount of data from a single memcache box to the client. If you want to transfer huge values this way, you still have to wait for individual chunks to arrive serially, so that might explain the timeouts you see - can you share the size of the values you're setting/fetching and the value of N you tried?
from mcrouter.
Thanks this clears it all up.
My timeouts were unrelated it turned out.
from mcrouter.
If anyone wants some fun, it turned out that the cached value for the view context of our blog posts was weighing in at 6.5mb. Each. This got rejected but I think once we turned on big value splitting it overloaded our cache servers. ;-)
from mcrouter.
The random suffix is needed for consistency - you can simply remove the original key and the chunks will be 'deleted' - there's no way to access them without knowing the random suffix. It also takes care of simultaneous sets, since only one key will win the race, and only its random suffix will be valid.
I have few questions about big value split threshold option:
- When key is deleted, I find it out that other related parts/chunks are not deleted. Is this expected behaviour or not?
- In case of simultaneous sets, last key that is set has references to its other parts/chunks which is ok. However, chunks that are related to previous sets are not deleted, but they should be?
mcrouter is 1.0 version (built using Dockerfile). Memcached version is 1.4.13.
mcrouter is started with this command:
# 5 bytes just in test purposes
mcrouter --big-value-split-threshold=5 --config-str='{"pools":{"A":{"servers":["127.0.0.1:5001"]}},"route":"PoolRoute|A"}' -p 5000
from mcrouter.
Related Issues (20)
- Simple replication model not working as expected HOT 1
- `mg` for WarmupRoute?
- CentOS 8 RPM/Build Support HOT 1
- Cannot build ubuntu20 docker file HOT 5
- Memcached server specification (TLS/SSL) documentation HOT 1
- Error to build docker image with docker file... HOT 1
- Last stable version
- Warmup related configuration incase we are running mcrouter and memcache on kubernetes
- Mcrouter adds MC_MSG_FLAG_BIG_VALUE flag but never removes it HOT 4
- AsyncSocketException: AsyncMcClient, type = Socket not open HOT 1
- unable to build new py3 thrift changes in upstream
- ubuntu 20.04 building issue HOT 1
- Failed to build on Ubuntu 22.04 HOT 1
- Addition of a pre-populated replica
- New Ubuntu package HOT 2
- Failed to build on ubuntu18.04
- How can I traverse a server in the pool? HOT 1
- Issues after memcached version upgrade
- does AllFastestRoute route consider miss as errors?
- folly build error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mcrouter.