foundatiofx / foundatio.redis Goto Github PK

I noticed some problems with the redis queue implementation.
I tried Redis Queue and Azure Service Bus Queue implementations so i cloud compare.

First Problem:
new RedisQueue<T>(o => o .ConnectionMultiplexer(ConnectionMultiplexer.Connect("redisConnection")) .Retries(1))
Won't do any retry at all. If you increase Retries to 2 it will do one retry so it seems the retry count is -1.
In Compare
return new AzureServiceBusQueue<T>(o => o .ConnectionString("asbConnection") .Retries(1))
Will retry once as expected.

Second Problem:
If i run the redis queue implementation with Retries(3) the Attempts property is wrong too:

Frist Run: Attempts -> 1
Frist Retry: Attempts -> 1
Second Retry: Attempts -> 2

In Compare to AzureServiceBus again:

Frist Run: Attempts -> 1
Frist Retry: Attempts -> 2
Second Retry: Attempts -> 3

I would like to to see an identical behavior.
Thanks

We have encountered an error when dequeue message from redis cluster while sending emails

[19:23:37 ERR] Retry error: Key has MOVED from Endpoint 10.20.0.14:9001 and hashslot 9229 but CommandFlags.NoRedirect was specified - redirect not followed for RPOPLPUSH {q:gaiaworks1-MailMessage}:in. IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=0,Free=32767,Min=4,Max=32767), Local-CPU: n/a
StackExchange.Redis.RedisServerException: Key has MOVED from Endpoint 10.20.0.14:9001 and hashslot 9229 but CommandFlags.NoRedirect was specified - redirect not followed for RPOPLPUSH {q:gaiaworks1-MailMessage}:in. IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=0,Free=32767,Min=4,Max=32767), Local-CPU: n/a
   at Foundatio.Queues.RedisQueue`1.<DequeueIdAsync>b__56_0() in C:\projects\foundatio-redis\src\Foundatio.Redis\Queues\RedisQueue.cs:line 331
   at Foundatio.Utility.Run.WithRetriesAsync[T](Func`1 action, Int32 maxAttempts, Nullable`1 retryInterval, CancellationToken cancellationToken, ILogger logger) in /app/src/Foundatio/Utility/Run.cs:line 41

In this lib, redis database are by default 0. In RedisCacheClientOptions, we can only supply a multiplexer instead of a specific StackExchange.Redis.IDatabase.

        private IDatabase Database => _options.ConnectionMultiplexer.GetDatabase();

In my case, I have several environments (staging, test, dev) all using the same redis server. Each env is assigned a different redis database number.

Hello,

When testing Publish/Subscribe with RedisMessageBus i've got some reliability issues.

If you kill down the Subscriber, and publish some messages with the client et then restart de Subscriber, messages are not processed.

Any idea on this?
Thanks you in advance.

Best Regards.
Yoann

        try {
            return await Run.WithRetriesAsync(async () => {
                return await Database.ListRightPopLeftPushAsync(_queueListName, _workListName).AnyContext();
            }, 3, TimeSpan.FromMilliseconds(100), linkedCancellationToken, _logger).AnyContext();
        } catch (Exception) {
            return RedisValue.Null;
        }

ListRightPopLeftPushAsync has default timeout of 5 seconds, in case of spike in Redis load which makes the action take more than 5 seconds (it's busy with other ops), this function will return RedisValue.Null while eventually the ListRightPopLeftPushAsync will actually be executed in Redis.

Since the item will be moved from "in" to "work" but this function will return null, the following code is not executed:

             await Run.WithRetriesAsync(() => Task.WhenAll(
            _cache.SetAsync(GetDequeuedTimeKey(value), now, wiTimeoutTtl),
            _cache.SetAsync(GetRenewedTimeKey(value), now, wiTimeoutTtl)

and the user handling function is not called.

So the item is now in "work" but only with "enqueued" value so maintenance will ignore it and it will stay in "work" forever without being handled.

Thanks.

The code is obviously opensource and I likely could figure it all out with some trouble. However I was wondering what is the "path" a message follows from beginning to end? For instance if there was stuck messages or a problem then how would I go about diagnosing this?

What does each general key pattern mean?

Also does it matter if every connection and every queue has maintenance turned on? When can it be turned off?
What does it even do?

After recent package update we started getting this error message, we tracked the issue down to a change done in RedisCacheClient.
All the read operations now use CommandFlags.PreferReplica (changed in this commit: 8ea1bf1)

Sometimes, the replica might not have the data when a read operation is performed right after write operation.
So this cache change breaks RedisQueue randomly.

My suggestion is to revert the change or allow injecting the required behavior.

According to the official documentation:
https://redis.io/commands/eval/

Important: to ensure the correct execution of scripts, both in standalone and clustered deployments, all names of keys that a script accesses must be explicitly provided as input key arguments. The script should only access keys whose names are given as input arguments. Scripts should never access keys with programmatically-generated names or based on the contents of data structures stored in the database.

Symptom that lead to looking into that:
Trying to remove ~10 000 keys by prefix silently failed.

After noticing an issue with CacheLockProvider where RenewAsync was causing locks to be released unexpectedly,
I followed the root cause to a line in RedisCacheClient:

The expiration argument should be sent to the script here in milliseconds and not in seconds.

i.e. the last argument should be: expires = expiresIn.Value.TotalMilliseconds

This is backed by the code in ReplaceIfEqual.lua - where the lua code calls the redis SET command with the PX option used for milliseconds.

I'm unable to push a proper pull request at this time. Perhaps this could be picked up by someone else.

Note that there's a list of additional Lua scripts in RedisCacheClient.LoadScriptsAsync that may or may not suffer from a similar issue, I didn't have the time to look into that.