antirez / disque-module Goto Github PK

https://nanomsg.github.io/nng/man/v1.1.0/index.html

right now you have to list all the jobs in one queue and call DELJOB for each of them to celar jobs on one queue. This is cumbersome and takes a lot of time if there is a lot of jobs.

One of the reasons Redis is so popular is because it's super easy to create a cluster with ElasticCache.
Unfortunately they don't support Redis Modules.
If we were to want disque to be popular, we'd have to make AWS support Redis Modules or disque specifically.

We have been using disque as a very simple queue on various projects and loved how simple it was to spin up a single instance and use it.

The requirement of spinning up a minimum three node redis cluster adds a lot of complexity especially in k8s based testing environments where the persistent volume support may not be present to enable this.

Please consider allowing it to work on a single node redis instance.

Thanks!

I implemented a partially-async replication patch for the original Disque project and back then put in a hopeful PR (antirez/disque#200). This was a vital feature for my use case (multiple datacenters) and something that I think has wider appeal and fits the project since Disque is designed as an "AP" system.

Is this type of replication something that could make its way to the Disque module? I am fully aware that the first priority is to reach feature / stability parity with the old project but I wanted to put this in early for consideration as it may simmer in some brains and spark ideas on even better ways to do especially multi-DC replication in split-brain situations.

P.S. Great to see Disque reborn and really appreciate the wonderful README!

For now Disque will consider a job replicated once there are N copies in th cluster. In case of multiple DCs in order to improve jobs safety especially if persistence is switched off, it is very important to also make sure there is at least one copy in a minimum number of AZ.

This could be implemented by having an AZ id configured in each Disque instance, up to AS 63 so that we can store then job replication info in little space using a bitmap.

ADDJOB will take an additional optional argument AZ to specify the minimum number of AZs that should have a copy. The replication acknowledge from nodes will include the AZ number so the receiving node can set the jobs AZ bits and understand if the replication was reached. If the first responders to arrive to N nodes as specified by REPLICATE where not in enough AZs, more nodes are dynamically added.

As an optimization Disque may want to remember the last AZ reported by each node to send the message to different AZs faster without using more copies than needed.

With streams being added in Redis 5, it seems like they include support for a lot of the same capabilities as disque is offering, especially through consumer groups. It would be nice to have a section in the documentation explaining what cases or needs might make you want to choose disque over streams, or vice versa.

Thanks for this awesome project.

At the beginning of the README.md it is stated:

IMPORTANT please read: This project for now does not accept pull requests, but feel free to open issues and discuss about bugs and features. PRs will be accepted later when a minimum level of stability is reached and I've the CLAs ready.

At the moment, this project is licensed with GPL v3.
Would you mind to shed more light into it on why a CLA is necessary here? No blaming or anything, I just want to understand it. Thank you. And enjoy your Sunday!

So the consumer can delay the requeing of failed job.