Comments (6)
For a producer, 1 connection is enough.
For the consumer, 3 connections are enough, one for the main process and two for the forks.
Well, if you are confident enough about your early statements, I highly encourage you to start your own project and propose it to the open source community, to be used in production-ready environments. I mean, go ahead! )
from redis-smq.
The number of connections for both the producer and the consumer is JUSTIFIED.
Let me make it clear.
By design, the RedisMQ is composed of many layers. Because we are talking about high performance (c), the main requirements for the requests to the Redis server of each layer are:
- Requests should be executed as fast as possible
- Requests from one layer should not block/delay the requests from other layers.
Considering the requirements, the best solution would be if each layer could operate on its own connections to Redis. Of course, new connections to Redis are made only when needed. For managing distributed locks, RedisSMQ uses the redlock package, which also maintains its own Redis connections (see https://www.npmjs.com/package/redlock).
Here is the minimum number of connections needed for a producer :
- A connection for publishing messages
- A connection for publishing statistical data (stats can be disabled)
- A connection for the scheduler
- A connection for the lock manager which is used by the scheduler
The number and the purpose of the connections used by a consumer instance can be verified the same way. Consumers are more complex than producers, and include more layers (gc, heartbeat, etc.).
RedisSMQ has been rigorously tested before being used in production at large scale for many projects that I developed. When benchmarking RedisSMQ, the numbers speak for themselves.
from redis-smq.
@intech In anyway, thank you for your feedback
from redis-smq.
Why create another product when it is possible to improve existing ones ;)
Thanks for the detailed explanation of the design.
I will do a detailed analysis to confirm or refute my statements about connections and locks in redis (single thread). I will also do a comparative test with keydb (multi-thread redis) and share the results in this issue.
from redis-smq.
If we are talking about high performance, then a large number of connections create locks at the redis level
Not true. First of all, a large number of connections is not the reason behind database locking issues. It does simply tell you that you have a problem somewhere in your code and that your should handle it, maybe a concurrency issue. In general, when dealing with locking issues (no matter at which level), looking at the number of requests (connections) is useless. Your should investigate the matter of your requests and figure out WHAT they are doing.
By the way, have you found any locks? I would be grateful if you could show me at least one issue in this regard.
from redis-smq.
You're welcome!
If you find any issue please feel free to report it.
from redis-smq.
Related Issues (20)
- LockManagerAcquireError: could not acquire a lock. HOT 3
- consumer for all queues HOT 2
- Managed redis connection keeps throwing error HOT 3
- [分享]可以让Redis消息队列更好的一篇文章 [Share] An article that can make Redis message queues better
- Feature request: Topic based routing HOT 3
- Cannot serialize `bigint` HOT 1
- Redis smq randomly throws an error that make app crashes HOT 2
- Support ioredis cluster configuration
- Upgrade to Redis 4 HOT 5
- Use instance based instead of a system wide configuration HOT 2
- Producer & Consumer seem to be ignoring config HOT 6
- How are messages stored in brokers in redis-smq? HOT 1
- Not working with docker compose HOT 5
- MaxListenersExceededWarning: Possible EventEmitter memory leak detected HOT 15
- [Question] How to avoid duplicated tasks? HOT 6
- LockManagerExtendError: Acquired lock could not be extended HOT 3
- random retryDelay HOT 4
- [Feature] Support option for a missing interval in the scheduler. HOT 4
- How to delete message in consumer HOT 4
- Error: ENOENT: no such file or directory, open '../lua/zpophgetrpush.lua HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from redis-smq.