Comments (8)
brayniac
commented on May 27, 2024
I'm unable to reproduce this on my laptop. The fact that memtier_benchmark fails to finish it's run is odd. Even if the server was not responding for some reason, it would ideally handle that and finish the run. Can you tell us more about the environment you're testing in? Might be helpful to know what OS/Version/Kernel/... and what hardware.
I use rpc-perf for my testing: https://github.com/iopsystems/rpc-perf
from pelikan.
bjzhjing
commented on May 27, 2024
@brayniac Thanks for your quick answer! Here is my environment info:
hw: Intel Alder Lake
os: Ubuntu 22.10
kernel: 5.19.0-28-generic
The output from memtier_benchmark side is as follows. There is one thread never stop until you manually kill it.
Writing results to stdout
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 4%, 0 secs] 20 threads: 1189585 ops, 3298300 (avg: 3298300) ops/sec, 158.46MB/sec (avg: 158.46MB/sec), 0.30 (avg:
[RUN #1 14%, 1 secs] 20 threads: 2361937 ops, 1175611 (avg: 1739413) ops/sec, 56.52MB/sec (avg: 83.60MB/sec), 0.85 (avg: 0
[RUN #1 24%, 2 secs] 20 threads: 3527287 ops, 1165214 (avg: 1495875) ops/sec, 55.94MB/sec (avg: 71.87MB/sec), 0.86 (avg: 0
[RUN #1 34%, 3 secs] 20 threads: 4709950 ops, 1182515 (avg: 1402549) ops/sec, 56.77MB/sec (avg: 67.37MB/sec), 0.84 (avg: 0
[RUN #1 44%, 4 secs] 20 threads: 5877782 ops, 1167694 (avg: 1348655) ops/sec, 56.07MB/sec (avg: 64.78MB/sec), 0.85 (avg: 0
[RUN #1 54%, 5 secs] 20 threads: 7091294 ops, 1213286 (avg: 1323388) ops/sec, 58.25MB/sec (avg: 63.56MB/sec), 0.82 (avg: 0
[RUN #1 64%, 6 secs] 20 threads: 8223382 ops, 1131880 (avg: 1293265) ops/sec, 54.34MB/sec (avg: 62.11MB/sec), 0.88 (avg: 0
[RUN #1 74%, 7 secs] 20 threads: 9387262 ops, 1163663 (avg: 1275649) ops/sec, 55.87MB/sec (avg: 61.26MB/sec), 0.86 (avg: 0
[RUN #1 84%, 8 secs] 20 threads: 10541236 ops, 1153834 (avg: 1261075) ops/sec, 55.41MB/sec (avg: 60.56MB/sec), 0.87 (avg: 0
[RUN #1 94%, 9 secs] 20 threads: 11667240 ops, 1125908 (avg: 1246631) ops/sec, 54.11MB/sec (avg: 59.87MB/sec), 0.89 (avg: 0
[RUN #1 100%, 9 secs] 13 threads: 12839641 ops, 1125908 (avg: 1283973) ops/sec, 54.11MB/sec (avg: 61.66MB/sec), 0.89 (avg:
[RUN #1 100%, 10 secs] 1 threads: 12854183 ops, 1125908 (avg: 1284879) ops/sec, 54.11MB/sec (avg: 61.71MB/sec), 0.89 (avg:
[RUN #1 100%, 10 secs] 1 threads: 12854183 ops, 1125908 (avg: 1284751) ops/sec, 54.11MB/sec (avg: 61.70MB/sec), 0.89 (avg:
[RUN #1 100%, 10 secs] 1 threads: 12854183 ops, 1125908 (avg: 1284622) ops/sec, 54.11MB/sec (avg: 61.69MB/sec), 0.89 (avg:
[RUN #1 100%, 10 secs] 1 threads: 12854183 ops, 1125908 (avg: 1284494) ops/sec, 54.11MB/sec (avg: 61.69MB/sec), 0.89 (avg:
[RUN #1 100%, 10 secs] 1 threads: 12854183 ops, 1125908 (avg: 1284365) ops/sec, 54.11MB/sec (avg: 61.68MB/sec), 0.89 (avg:
[RUN #1 100%, 10 secs] 1 threads: 12854183 ops, 1125908 (avg: 1284237) ops/sec, 54.11MB/sec (avg: 61.67MB/sec), 0.89 (avg:
[RUN #1 100%, 10 secs] 1 threads: 12854183 ops, 1125908 (avg: 1284109) ops/sec, 54.11MB/sec (avg: 61.67MB/sec), 0.89 (avg:
Please be sure to set config/segcache with 6 worker threads or higher, and run memtier_benchmark with 20 threads or higher.
Additionally, I've tried ubuntu 20.04 with kernel 5.4, the problem is also able to be reproduced.
from pelikan.
bjzhjing
commented on May 27, 2024
One more thing that the performance test was previously running on host directly, both pelikan and memtier_benchmark is installed on host. But when I run pelikan/pelikan_segcache_rs in docker image while memtier_benchmark is still on host, the issue disappears.
from pelikan.
hderms
commented on May 27, 2024
@bjzhjing does this happen every time?
from pelikan.
brayniac
commented on May 27, 2024
Is it possible you're running out of available open file handles? It's
strange that memtier_benchmark would hang like that. But it's hard for me
to believe this is a Pelikan issue and not just a system configuration and
test setup issue. I think memtier_benchmark needs to be made more resilient
too.
For the record, I tested with even high number of Segcache workers and
memtier_benchmark threads than you and the test completed just fine. That's
on MacOS though. But we have run Segcache on production systems handling
10k concurrent connections without any observed issues.
You probably want to check `ulimit -n` with the numbers you provided, I
suspect that value is going to be too low. Default is 50 clients per thread
times 20 memtier_benchmark threads is 1000.
…On Thu, Feb 2, 2023, 4:42 AM bjzhjing ***@***.***> wrote:
One more thing that the performance test was previously running on host
directly, both pelikan and memtier_benchmark is installed on host. But when
I run pelikan/pelikan_segcache_rs in docker image while memtier_benchmark
is still on host, the issue disappears.
—
Reply to this email directly, view it on GitHub
<#33 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACP6HFYRTWIIUNN3S7IQKLLWVOTSVANCNFSM6AAAAAAUNKQYUY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
from pelikan.
hderms
commented on May 27, 2024
@bjzhjing what specific git SHA are you using? I am going to try to reproduce this as well
from pelikan.
hderms
commented on May 27, 2024
@bjzhjing I am quite certain at this point that @brayniac hit the nail on the head. I dropped my ulimit to 900 ulimit -n 900 before running both segcache and memtier and I can reproduce a hang where it fails to quit out after 10 seconds have elapsed. Everything worked properly with the default ulimit on my system (1024). I then set the ulimit back to 1024 and restarted both processes and it succeeded again.
I haven't bothered to see if it's required that the ulimit be lower than necessary for both segcache and memtier in order to reproduce the hang, but I can't reproduce it without segcache being given too low of a ulimit.
from pelikan.
bjzhjing
commented on May 27, 2024
@brayniac @hderms Yeah, you're right. It's due to the ulimit value. The default is 1024 on my system, after change it to 4096, the test runs well. I ignored this setting for no message indicates that. Thanks a bunch!
from pelikan.
Related Issues (16)
- Publish on crates.io HOT 5
- Add basic Redis compatible backend
- Add storage library to support datastructures
- Add support for ssd/nvme storage
- Add support for tiered storage
- Add support for drop-in replacement of guava HOT 4
- Add support for unix domain sockets
- Add log message when hash power is too low HOT 3
- `Set` command option's like 'EX 1000' and NX failing to parse
- Add fuzzer for RESP protocol HOT 1
- Resp protocol parser crashes when given a command ending with a newline HOT 1
- Question about segcache eviction policy HOT 22
- Segcache integration test is flaky within MacOS CI HOT 1
- Handle `SIGINT` in segcache HOT 6
- Building Dockerfile results in missing protoc error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pelikan.