• External Server tests on PRs (.github/workflows/external.yml)
• Change parts in Issue templates

This is a true gem :). The proposal was originally brought up by @touitou-dan and below is a verbatim copy of redis/redis#12489

The following is a proposal to accelerate Redis performance by:

Improving io-thread efficiency by totally offloading network layer from main thread.
Reducing load on main thread remaining functionality by moving memory management and response formatting to the io-threads.
Amortizing memory access latency by batching dictionary searches.
We believe than by implementing these steps, a single Redis instance running on a multicore machine will deliver more than 1 million rps, up from under 400K rps as it is today.

Introduction

Io-threads were added to version 6.0 in order to split the network and some part of the application representation layer processing load among two or more cores. Io-threads can be used for both incoming requests and outgoing responses. When used on incoming traffic, io-threads not only read from the socket but also attempt to parse the first incoming request in the input buffer. In both directions, the main thread operates in a similar way, it balances the io tasks equally between the io-threads and itself, executes its part of the tasks and waits for others to complete their part before processing commands. This fan-out/fan-in approach keeps the solution simple has it requires no complex synchronization except from a barrier between the main and io threads.
We measured the performance of Redis (version 7.0) with and without io-threads. All tests were performed on an EC2 Graviton 3 instance with 8 cores (r7g.2xl) with no replica and no TLS. In this test we first populated the DB with 3 million keys of size 512 bytes and sent GET/SET requests from 500 clients. The GET and SET distribution was 80 and 20% respectively. We measured the following performance numbers:

Without io-threads 205K rps
With 6 additional io-threads (“—io-threads 7”) 295K rps
With 6 additional io-threads doing also read 390K rps
When analyzing the performance we found the following issues:
a. Underutilized cores - despite the 2x performance acceleration io-threads provide, Redis main thread still spends only 57% (processPendingCommandAndInputBuffer) of time executing commands. This implies that 1. The io-threads are practically idle 57% of the time and that 2. the main thread is spending 43% of the time executing io related functionality that could and should be executed by the io-threads. In addition, out of the 57% spent on executing commands, more than 9 percent (addReplyBulk ) are spent on translating objects into responses.
b. Memory management – Redis main thread spends more than 7% freeing object that have been allocated mostly by the io-threads. Such discordance between allocators and freers may cause lock contention on the jemalloc arenas and reduce efficiency.
c. Memory pattern access – Redis dictionary is a straightforward but inefficient chained hash implementation. While traversing the hash linked lists, every access to either a dictEntry structure, pointer to key or a value object requires with high probability an expensive external memory access. In our tests we found that the main thread spent more that 26% on dictionary related operations.

Our suggestions:

Io-threads

We suggest to totally offload all network layer functionality from main thread to io-threads. Our preferred approach would be to divide the client layer into two halves. The first half, handled by the io-thread will be the “stream layer” which includes socket layer as well as parsing requests and formatting responses while the second part, handled by main thread, maintains the client execution’s state (blocked, watching, subscribing etc). Based on this, io-threads handle read/write/epoll on the sockets, parse and allocate commands and append the ready to be executed commands on one or more queues. The main thread extract commands from the queues, executes them in the client context and appends responses on the queues which the io-threads extract, format and transmit on the sockets. In addition to the standard Redis commands and responses, internal control commands will be exchange between io-threads and main thread such as creating and erasing client states, pausing read from client sockets, requests to free previously allocated memory etc..

Dictionary memory access

Redis de facto executes commands in batches. We suggest that before executing a batch of commands we must ensure that all memory locations needed for dictionary operations will be find in the (L1/2/3) caches and avoid the latency associated with external memory accesses. This is done by searching in the dictionary all keys from a batch of commands before executing them. Searching the dictionary with more than on key at a time, when using prefetch instructions properly, allows to amortize memory access latency. From our experience, up to 65 % of the time spent on dictionary operations can be reduced this way.

Alternative solutions

An alternative approach to increase parallelism would be to allow main and io threads run in parallel on an unmodified client layer and prevent conflicting accesses between the io and main thread with locks. Such approach, while theoretically simpler, may require considerable testing to ensure consistency as well as avoiding lock related issues such as contention and deadlocks.

Dictionary inefficiencies can be solved by replacing the hash table implementation with a more "cache friendly" one that amortizes memory access by storing entire buckets in one or more adjacent cache lines. This approach has the advantage of being more "neighbor friendly" as it issues considerably less requests to the memory channels. However since Redis dictionary deliver a non standard programmatic interface ("key to dictEntry" mapping rather than "key to value" mapping) , replacing the hash implementation requires a much bigger coding and testing effort than the proposed alternative.

Are you going to "fork" them too?

Re-sharding in OSS Redis clusters is a risky operation, lacking both high availability and eventual consistency in the design. This is a pain point for many users. Mitigations were previously discussed (see pull request redis/redis#10517), and I believe it's important to resurrect this conversation within this fork. These mitigations could provide much-needed relief while we work towards a more robust long-term solution. I'd like to hear the community's thoughts on the feasibility of these mitigations and how they could benefit the users.

@zuiderkwast @soloestoy @madolson

The problem/use-case that the feature addresses
Currently, a key is deleted when the set is empty. Distinguishing between an empty set and a key that does not exist offers many advantages. For example, if we store SQL results in a set, an empty set can represent that a query was executed but the result was empty, whereas null might imply that we need to make a database call to store in the set.

Description of the feature
Without breaking any backward compatibilities we scan support the Empty set with set of new commands and arguments

Write commands
SADD key
Existing SADD would be modified to make members as optional, invoking SADD with only key will create an empty set.

SPOP key [count] MT → Additional argument
Add new optional argument to retain the SET if no element exist.

SREMMT key member [member ...] → [New Command]
Same as SREM would keep the empty set once all members are removed.

SMOVEMT , SINTERSTOREMT, SDIFFSTOREMT, and SUNIONSTOREMT would be implemented as followups if needed. One can delete the empty set using DEL command.

Read commands
SCARD would still return 0 for empty set. To distinguish empty set from a non existing key one can use EXISTS followed by SCARD. Same can be approach for the SISMEMBER and SMISMEMBER.

SMEMBERSMT key → [New Command]
Similar to SMEMBERS but will return null when key does not exist. This command can be achieved by using EXISTS + SMEMBERS but would be supported for ease of use.

Alternatives you've considered
Provide a new param to change the default behavior.

The problem/use-case that the feature addresses

Currently, some clients rely on a single-node perspective to obtain cluster topology, which may not accurately reflect the entire cluster's state. Ideally, clients should query multiple nodes and compare their topology views to determine the most consistent representation. To facilitate efficient comparison on the client side, if Valkey can offer a deterministic view where nodes with identical views return identical outputs, clients could easily hash and compare these outputs. ATM, certain variables in the CLUSTER SHARDS command, such as "replication-offset," may differ between nodes even when they share the same view, so the client must parse the view and filter out each view by itself. Furthermore, in the current output of CLUSTER SHARDS (or CLUSTER SLOTS), nodes with identical views may present the replicas of specific slots in differing orders. This necessitates clients to parse and sort the outputs for accurate view comparison.

Description of the feature

In order to achieve deterministic output, 2 changes are suggested:

1. Adding a filter option to the CLUSTER SHARDS command. This enhancement would enable clients to selectively filter out irrelevant fields for their specific use cases, such as non-deterministic fields, nodes in 'fail' state that irrelevant for the client functionality, or other fields that the client doesn't need. It would enable hash-based comparison of views without the need for parsing the output, while also potentially reducing the size of each node's output.

2. Sorting the replica order in the output.

Blocking commands complicate multiplexing several logical connections over the same physical connection, and in general are a perf hassle. There are 2 existing suggestions for ways to overcome this issue -

redis/redis#12873
redis/redis#12716

either one could allow client writers to safely multiplex several client objects over the same physical connection, and help users get the most out of their machines.

Ideally all of the description should be:

1. Full sentences with capitalization
2. Include punctuation.

Hi,

First of all, thanks for your fork and for your trials to maintaining it.
I am one of the Redis DBAs (w. internal cloud service), who have recently being interested in opensource ecosystem.

because i'm newbie in opensource, I could just write some short commits about improving usability.
https://github.com/redis/redis/pulls/LiiNen

The point is, could I wrote both issues or PR in redis and placeholderkv?
If so, although both are written by me, I just worry about that License's of these two are different.

Best regards,

P.S. I will be glad if you (or with AWS?) could maintain this "OPENSOURCE" kv store
oriented from redis, which "WAS" opensource. thks.

It would be interesting to learn what kind of features you guys are planning to add after a fully functional Redis fork? Or would it stay in maintainance mode for some time until the future roadmap is officially announced.

The command INFO includes the following lines:

# Server
redis_version:255.255.255
redis_git_sha1:daadbca2
redis_git_dirty:1
redis_build_id:187c43196ce2ead2
redis_mode:standalone

Clients use this information to find out which version the server is running and from this they can conclude which features it supports.

The question is whether API compatibility justifies using a trademark name in the output of the command.

We have some alternatives:

1. Change "redis_version" to "server_version" (etc.) and also add a "server_name:newname"
2. Change the prefix, e.g. "newname_version", etc.
3. Keep the existing output.

Will the valkey servers be fully compatible with the programming language sdks used by redis? Or something else will be developed for valkey?

It would be nice to finally de-crapify and de-egoify the project fully. Just make a new primary version and go backwards incompatible.

I'm fairly certain the blanket "license change" is also illegal since they didn't get permission from every copyright holder to rip their code apart from the license (also why linux can never change from GPL2), but if they don't care, then we don't have to care about un-licensing all their code either. Evil goes both ways I guess (that article has some timeline inaccuracies and rationale inaccuracies and leaves out multiple other key players, but it's mostly correct in how redis was "stolen" by motivated exploiters over the years).

Wishlist

• #36
• remove sentinel and cluster (they will never work reliably because they are fundamentally flawed in their design. notice how nothing else in the world uses them or their design patterns? at least clickhouse liked zookeeper so they rewrote it in C++ for their own usage)
  • overall it's more stable going with separation-of-concerns and using consistent-hash-replication sharding proxies in front of these things anyway, though maybe some CRDTs will need to get involved
• port all tcl scripts to something better (it's 2024 not 1994)
  • new code robots can greatly help with this I imagine
• use clang-format for all the code (it's 2024 not 1994)
  • clang-format -style="{BasedOnStyle: llvm, IndentWidth: 4, AllowShortFunctionsOnASingleLine: None, KeepEmptyLinesAtTheStartOfBlocks: false, InsertBraces: true; InsertNewlineAtEOF: true}"
• stop supporting weird platforms because it's 2024 and nobody cares about big endian or 32 bit systems anymore (it causes unnecessary maintenance overhead for refactoring and building new features)
• stop using CRC everywhere. again, it's 2024 not 1994. Nobody should be using CRC anymore because it's designed for 1970s serial interfaces. We have much better systems now.
  • like, this thing is complete amateur garbage what are people even doing anymore: https://github.com/placeholderkv/placeholderkv/blob/unstable/src/crc16_slottable.h
  • attempting to use designed-to-be-slow CRC running at 300 MB/s (or 1.5 GB/s with my improvements I spent a year coming up with 10 years ago) to blockingly validate multi-gigabyte files is just computational malpractice when things like xxh3 or xxh128 exists running at 30 GB/s.
• fix the hand written, hand parsed, self-rewriting config file system abomination (nothing else in the world does this, and it caused a wormable exploit in the past so should never be trusted. it only exists because "lol its a cool trick!!!" unprofessional insecure programming)
• stop using hand written Makefiles and use CMake instead (again, it's 2024 not 1994)
  • I actually spent a few hundred hours making jemalloc compatible with CMake in 2017-2019 then when I aksed FB if they could pay for my contributions they said "uh, we can send you a t-shirt?" then they never even sent the t-shirt (a month later, FB paid a $5 billion government fine for their typical FB-style business practices, but they sure don't have any money to pay developers... go figure?)
• convert the info/status output to regular readable JSON instead of hand-written output formats requiring hand-written parsers to read
• continue to refactor and improve a lot of the data structures from their unrefined CS101 origins

there's probably a couple other dozen things too, but that's my long-term gripe list I'm really surprised most people haven't been motivated to see the problems and fix them yet for long term project reliability and continuity.

Conclusion

I fixed all these problems (and more!) in my own rewrite a couple years ago (just imagine you put luajit, sqlite, erlang, redis, and memcached in a blender then a new modern high-performance multi-core secure in-memory cache system popped out), but I tried to sell it instead of giving away thousands of hours of work across 5+ years for free, so it never got any traction: https://carrierdb.cloud/

Provenance

i used to do stuff here but the project preferred to remain computationally and culturally conservative instead of embracing the future (except for selling out for profit at the expense of community and collaboration and user experience, of course) :(

Work I created which the project sometimes fought against but eventually included anyway:

• entire module system was my own design and implementation, but the project fought against it until later they decided it could be used for profit
• geo commands (though they aren't complete and should really use h3 instead of geohash, but nobody seems to understand enough to maintain it over the past 10 years)
• JSON experiments (not sure what happened with those, but my redis replacement server can switch between output formats of redis, memcached, JSON, and even python (only format with actual set syntax in the output), and it also supports unlimited nesting of any data type each with their own nested expiration values too)
• massive memory improvements to the lists structures (further efficiency improvements are in my redis replacement server too along with improving all the other data structures as well) — this was inspired by the custom changes to redis Twitter was running internally over thousands of servers totaling terabytes of RAM (which was a lot in 2015), but the twitter devs refused to release their improvements because "we can't get legal to clear it," so I re-built their ideas in an even better version from scratch (since then, millions of servers have benefited from these efficiency improvements for free, so thanks to me i guess)
• crc speed improvements which took a couple hundred hours of on-and-off experiments over a year to finally get the logic working because redis uses a random weird non-standard CRC64 implementation for some reason so no other libraries are compatible with it unless you do extensive custom workarounds all over the place.

placeholderkv$ rg "Matt Stancliff"                                                                                       on unstable
src/quicklist.c
3: * Copyright (c) 2014, Matt Stancliff <matt@genges.com>

src/geo.c
2: * Copyright (c) 2014, Matt Stancliff <matt@genges.com>.

src/geohash_helper.c
3: * Copyright (c) 2014, Matt Stancliff <matt@genges.com>.

src/crcspeed.h
1:/* Copyright (c) 2014, Matt Stancliff <matt@genges.com>

src/quicklist.h
3: * Copyright (c) 2014, Matt Stancliff <matt@genges.com>

src/geohash_helper.h
3: * Copyright (c) 2014, Matt Stancliff <matt@genges.com>.

src/geohash.h
3: * Copyright (c) 2014, Matt Stancliff <matt@genges.com>.

src/zmalloc.c
865: * 3) Was modified for Redis by Matt Stancliff.

src/crcspeed.c
4: * Modifications by Matt Stancliff <matt@genges.com>:

src/geohash.c
3: * Copyright (c) 2014, Matt Stancliff <matt@genges.com>.

src/crc64.c
1:/* Copyright (c) 2014, Matt Stancliff <matt@genges.com>

deps/hiredis/hiredis.c
4: * Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,

deps/hiredis/net.c
5: * Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,

deps/hiredis/net.h
5: * Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,

deps/hiredis/fuzzing/format_command_fuzzer.c
4: * Copyright (c) 2020, Matt Stancliff <matt at genges dot com>,

deps/hiredis/hiredis.h
4: * Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,

deps/hiredis/CHANGELOG.md
511:* Fix tests when assert() undefined (Keith Bennett, Matt Stancliff)

Per discussion in #43, we should consider tackling (more) compat issues holistically in RC2.

IMO, I think we should break as much as we like in RC2 the first major release and only, which includes changing the server ident from "redis" to "valkey", replacing "master/slave" with "primary/replica", etc in all user facing areas. All these changes should be gated upon a compat knob, which could be binary (on vs off) or a bit more granular (strict/loose/none) but shouldn't be more granular than that. After achieving this in RC2 the first major release, I think we should agree on a timeline to completely remove this compat layer (in 1-2 years or major releases, ideally).

I don't have the details yet, such as whether this compat knob is compile time, build time or packaging time, and what does "loose" mean, etc. I am curious to hear the community's thoughts on the high level direction.

@valkey-io/core-team

DESCRIPTION

Introduce cross version/cross fork integration testing infrastructure. With the compatibility version release and planned new major version release, it will be good to improve the testing/release certification process. This will help Valkey to prepare for release(s) more confidently and avoid pain for user(s) during migration/upgrade(s).

Example Scenario:

Issue: redis/redis#12685

Redis 7.2 introduced cluster bus message extensions feature by default and it caused failure of engine upgrade from older version (i.e. Redis 6.2 or lower) due to message broadcasted from engine running 7.2 not being compatible in older versions.

PR to fix the issue during upgrade: #52

I've gone ahead and added a DCO check for our version. We want to make sure we aren't unintentionally pulling in code from either redict (which is LGPL) or Redis (which is now RSAL and SSPL).

All that is required is for the commit to contain Signed-off-by: Full Name <email>. Starting this thread to raise any concerns about the issue.

The problem/use-case that the feature addresses

Today clients find out that a topology change happened only after the fact - either by periodically querying CLUSTER NODES/SLOTS/SHARDS or by receiving MOVED/ASK errors. When a client finds out that a topology change happened by receiving an error, the client needs to call CLUSTER NODES/SLOTS/SHARDS in order to get the new cluster topology, which might be slow on large, fragmented clusters.

Description of the feature

Using RESP3 push messages, nodes might send clients updates on topology changes or slot migrations, with all the relevant information. This means that

1. clients are updated during the change, not after the fact
2. clients could receive only the relevant info (slot X was moved from A to B), instead of having to query the whole topology. this is both more economical in regards to network traffic, and doesn't block the server for slow calls.

Alternatives you've considered

I believe this can be easily implemented for slot migration, but topology changes will probably be harder - which node should inform the client about the new nodes? an alternative might be to just inform the client on epoch changes, and let the client query the current topology in the usual way.

Whenever we introduce new data structures or new encodings, we need to adjust the RDB's version and encoding format. Internally in the server, this is not an issue. However, changes to RDB create a substantial adaption workload for external tools.

For example, some data migration tools like redis-shake, which needs to parse the RDB file during full synchronization, has to put effort into adapting to new versions whenever there are changes in the RDB (RDB version as well as many changes in storage structures). The tool needs to parse RDB to extract key-value pairs and transform them into restore commands before sending them to the target instance.

Also, there are many detailed issues to be addressed, such as the restore command's parameters not exceeding 500MB (due to proto-max-bulk-len limitation), meaning that when dealing with larger payloads, it's necessary to analyze the specific storage format for splitting, and then reverse-engineer it back into commands for replay (for example, a large hash would be converted into multiple HSET commands).

Furthermore, instances on old versions cannot parse RDB from new versions and cannot use the restore command for data migration (some special scenarios may require rolling back to a previous version by using data migration tools).

To solve the problems and to ensure that migration tools are not affected by changes in RDB format, we can add a new method for full synchronization: using the AOF file (where AOF specifically refers to a collection of commands, not using an RDB preamble). While full synchronization between master and replica still uses the RDB file, data migration tools could declare the file format they wish to use during full synchronization via the REPLCONF command. By doing so, data migration tools can simply forward commands without needing to parse RDB, allowing the full synchronization data to be directly passed through to the target instance, thus simplifying the adaptation work.

make test results in You need tcl 8.5 or newer in order to run the Redis test.

It would be great if the tests were compiled as go program(s) instead of tcl scripts.

> hello 3
1# "server" => "redis"
2# "version" => "7.2.4"
(...)

I think we should change it anyway, immediately in our first compatibility version. It is very low risk that any client depends on this value, since it's static.

For a better decision, what do other forks and redis-compatible services (AWS, Alibaba, ...) return for this field?

We can support both Makefile and CMake compilation methods. This requirement is also mentioned in #17.

• https://cmake.org/
• https://code.visualstudio.com/docs/cpp/cmake-linux
• https://www.jetbrains.com/help/clion/quick-cmake-tutorial.html

The problem/use-case that the feature addresses

Implement a scan cursor that can consistently scan the entire cluster, instead of today which requires individually targeting each node and sending it an individual SCAN command. This can also break, as slots can be migrated off the node and failovers can happen.

Description of the feature

Implement a consistent cluster scan with semantics like:

CSCAN <cluster cursor> [MATCH pattern] [COUNT count] [TYPE type]

The cluster cursor would be marked as NOT_KEY, but would be hashed like all other keys by the clients so that they would be routed to the next node. The cursor would contain a component that includes a hashtag, to represent the slot it's currently scanning.

the format of the cursor would be:

<version>-{hashtag}-<slot db id>

Alternatives you've considered

Extending the existing SCAN cursor to support scanning a specific slot, something like:

SCAN <cluster cursor> [SLOT X] [MATCH pattern] [COUNT count] [TYPE type]

This would require the end user to specific a specific slot, and the scan command would parse just that specific slot.

Additional information

See redis/redis#2702.

In

We need to include our responsible disclosure agreement as well as our maintenance plan. I think this is something worth figuring out now. I would like to suggest we introduce a policy of maintaining all versions for at least 3 years. This will be a bit different from Redis, as we had some weird number of major/minor thing. The time will start when the release is first GA'd, so we will commit to maintaining our compatibility fork for 3 years.

Hi, I'm one of the people working on the Redict fork, and we would like to merge efforts if possible. Bringing the discussion here from #8 (and various private conversations) where there is some additional context.

The only thing we're really committed to is the name and the use of the LGPL license. If the folks working on this fork can get on board with that, then it does not make sense to split the community in two, but we are committed to the copyleft approach so we will need to get consensus on this to merge forks. The rationale behind LGPL has been explained in a few different places, but I've summarized it here.

As far as practical matters are concerned, there is the matter of infrastructure and communication. We're using Codeberg rather than GitHub right now, which should be very familiar to GitHub users and has the advantage of being free software, which we are strongly in support of using. I hope you're willing to give it a shot, but I don't think it's a hill worth dying on. Discord on the other hand, I can't speak for everyone involved in Redict but I think it's a very hard sell. We're using IRC right now, but we could be convinced to use Matrix, which is a bit more user friendly.

Is PlaceholderKV looking to continue current versioning and whatever is next after 7.2.4 for the next release? Or start over from 1.0.0?

Also, what is the versioning scheme? The previous project was nominally SemVer but there was definitely some non-adherence to the spec. If SemVer is the methodology, I'd love to see an explicit definition of the public API.

There are several features on our roadmap that assume that we will develop some form of locking on an individual key. I think there is only one commonly accepted approach to achieving this, which is to mark a key as "in use" and block commands that attempt to mutate that key. Commands may either take a "shared lock", multiple different places can take a "shared lock" on the key, or an "exclusive lock", only one such lock can be granted at a given time.

This issue is just a high level summary of all the known issues related to key locking. Most of this is derived from redis/redis#7372 (comment). Re-posting verbatim here:

Even if the command is not explicitly writing, there are several cases where data is modified when reading. These include: PFCOUNT and also any data structure where the format can be internally changed - like hash table, skiplist, compressed list encoding which can be in the process of rehashing or other data transformation. These modifications must be prevented in both main and background threads.
There are significant complexities with MULTI/EXEC. If a MULTI/EXEC hit’s a blocked key, it must either:
synchronously pause the Redis main thread (waiting for the background thread to release the lock) - this also comes with complexity regarding completing/joining the background command while the main thread is paused.
pre-examine the MULTI/EXEC to determine if any locked keys will be used. This also comes with the challenge of tracking keys across databases if SWAPDB and/or SELECT is used in the MULTI/EXEC block.
block all MULTI/EXEC commands if any async command is running.
There are significant challenges with LUA. LUA is not forced to declare keys used. Furthermore, even if a LUA script does declare which keys are touched, the declaration does not indicate which DB the key is associated with. There is no mechanism to determine keys by inspection (as is possible with MULTI/EXEC). The safest option for LUA might be to block any LUA script if any async command is running. Throwing an error is not an option as it breaks atomicity guarantees.
There is a conflict with FLUSHDB/FLUSHALL. These keyless commands would need to be blocked until no background thread was running.
There is the possibility of interaction with EXPIRE and EVICT processing. Deep dive is necessary here to understand the issues with each.
Client disconnect (on the main thread) needs to interact with async processing in a background thread.
BRPOPLPUSH is a special can of worms and needs to be carefully considered. Consider BRPOPLPUSH FROM TO. Later, while a background thread has “TO” locked, another client performs an LPUSH FROM. The LPUSH (to a different key) would need to be blocked OR the BRPOPLPUSH would need to be further blocked.
Should MONITOR show the executed command at the beginning of processing? or the end?
I know you’re currently looking at background reads, but if the slow command needs to perform a write, the logical lock must also block readers on the main thread, not just writers.
If the background command is a write command, unlocking of the key needs to be coordinated with replication.
In addition, an important part of this proposal involves the blocking of clients until a key is unlocked. The existing blocking mechanism (for blocking commands like BRPOPLPUSH) is a good candidate for re-work at this juncture.

The existing blocking mechanism works as follows:

The blocking command is processed normally through processCommand().
In the command’s handler, the command is removed from the client’s normal cmd/argv/argc fields and copied over to custom fields for the command. (The normal fields are cleared.)
When the command is unblocked, there is a completely separate code path (independent from processCommand()) which is used to finish execution of the blocking command.
Not only does a separate code path require independent maintenance, but can lead to subtle differences in the way commands function if blocked. (Hint: IIRC, MONITORing a BRPOPLPUSH will look subtly different if it blocks or not.)
Instead, changing the blocking mechanism will better support this new use-case (and potentially others).

When a client is blocked (for any reason), leave the cmd/argv/argc attached on the client as normal.
When the client is unblocked, pass it back through processCommand() to execute as normal. This might result in the client being re-blocked for another reason.
Some care must be taken to ensure that MONITOR works as expected and only logs the 1st attempt.
This approach eliminates a bunch of blocking command fields on the client struct. It eliminates a separate code path for finishing these commands. It provides a standard approach to blocking clients.
Note that some adjustment of timeout might be necessary when re-blocking.
Example: A BRPOPLPUSH is blocked waiting on something to pop. A LPUSH is performed which unblocks the BRPOPLPUSH. We attempt to execute BRPOPLPUSH normally by sending it through processCommand(). However it is discovered that the destination key is locked. The BRPOPLPUSH client is re-blocked. Later, when the target key is unlocked, we again send BRPOPLPUSH through processCommand() and this time it succeeds.

Example: “MSET k1 v1 k2 v2” is performed. It is recognized that k1 is locked. The client is blocked. When k1 is unlocked, the MSET command is passed back through processCommand(). It is determined that now, k2 is locked. The client is blocked a second time. When k2 is unlocked, the command is sent through processCommand() a third time, and may execute normally.

Old ref: redis/redis#11396

Primary and replica, are these the preferred terms?

The current situation is master/replica, with slave kept as an alias of replica. Some commands like ROLE still returns "master" and "slave".

So far "master" was accepted, so there aren't yet any aliases for "master". Shall we first introduce aliases and deprecate master in our first release, then in our next release (not too far in the future) do all the breaking changes?

1. Add aliases for master (not a breaking change)
2. Delete master and slave terminology (breaking change)

Occurrences of "master" where we can add "primary" and make "master" an alias:

• CLIENT KILL TYPE MASTER
• CLIENT LIST TYPE MASTER
• SENTINEL GET-MASTER-ADDR-BY-NAME

Breaking changes:

• Remove all aliases "slave"
• Replace all remaining occurrences of "slave" and "master" with the new terms. This includes the return value of the ROLE command.
• CLUSTER SHARDS ("role" => "master", in the reply)

We have a requirement that we need to get a notification on changes to a Redis data structure. Based on my research I found out that I can use Redis key space notifications for doing the same. However, Redis key space notifications send the events to Redis pub/sub channel which is fire and forget i.e once the clients lose the connections all the events till the connection is up again are lost.

Redis streams solve this problem. Also, I want to use consumer group feature of Redis streams. So is there any way that Redis key space notifications can be pushed to Redis streams instead of Redis pub/sub channel?

One of the feature request which I was interested in solving in the past, so filing an issue to have more discussion on this
Ref: redis/redis#5766

RESP represents Redis Serialization Protocol.
In the future, maybe we should consider a new protocol to replace it,
but the format could support back compatible with redis or it will be break change

Get rid of the Redis name, except where it's needed for API compatibility. This can be done once we have a name.

Some of the below changes have a low risk of breaking e.g. script parsing logs or INFO output, but I'd say those are acceptable.

• ☑️ Make variables REDIS_CFLAGS and REDIS_LDFLAGS. Done in 3863227. They were renamed to SERVER_*.

• #148

• #204

• #207

• Occurrences of "Redis" in command JSON files (src/commands/*.json) #35

• #41

  # Server
  redis_version:255.255.255
  redis_git_sha1:daadbca2
  redis_git_dirty:1
  redis_build_id:187c43196ce2ead2
  redis_mode:standalone
  

  We have two options:

  1. Change to "server_version" (etc.) and also add a "server_name:newname"
  2. Change the prefix, e.g. "newname_version", etc.

• #61

  > hello 3
  1# "server" => "redis"
  2# "version" => "7.2.4"
  (...)
  

• #280

• #144

• #137

• #134

• #135

• #136

• Rename the binaries to redis-server, redis-sentinel, redis-check-rdb, redis-check-aof, redis-cli, redis-benchmark and their corresponding .c and .h files. Done in 3863227.

• #153

• #147

• #146

• #321

• #354

The problem/use-case that the feature addresses

Ability to deploy Valkey on a docker environment, or replace existing redis deployments

Description of the feature

Automated docker builds for each release and/or commit, pushed ghcr.io and/or docker hub's registry under the open source program for increased pull limits

This issue was first raised in Redis community redis/redis#11780, it is a major miss from Redis 7.0, but haven't got a solution. I believe we still need to discuss this issue in this new project, perhaps as a part of new cluster architecture.

The problem:

Let's say:

• a slot, which includes key1, is migrated from node A to node B, operated by client1
• in node A a function myfunc is already loaded, but in node B it is not

Now client2, that used to handle key1 in node A, sends FCALL myfunc 1 key1 to node A, which will be redirected to node B, will then get an unexpected error.
Or if node B also has a function myfunc but it does different things, client2 may trigger unexpected results.

Some thoughts:

1.Should functions be migrated together with slots?
Maybe not. This may overwrite the libraries already loaded in the nodes importing slots. But we may set a version to each library so that we can always choose to keep the latest one.

2.Can functions be broadcasted to all nodes?
May cause confict between nodes. And when adding new nodes we need to have a full synchronization.

Yet another gem that we should seriously consider for our new project. The original proposal (and the thread) is too long so I am going to leave it at redis/redis#10933.

Note that this feature will be orthogonal to how cluster topology management is done.

Hello,

Is it an official fork from AWS Elasticache Team?

I’m revisiting the feature proposal we discussed in redis/redis#10472, which aims at providing metrics at the slot level. Despite the substantial effort and detailed discussions back then, we didn’t land this feature. I believe it’s worth reconsidering, given the potential benefits and previous interest.

@kyle-yh-kim @zuiderkwast @madolson

We shall also keep redis for backward compatibility.

Now that Valkey is beginning to set its own course, wonder if Valkey can be enhanced to work as general data store, rather than just in-memory data store? It will widen the use-case for Valkey and remove those memory constraint.

Discussed in https://github.com/orgs/valkey-io/discussions/125

Scan is supposed to provide the following guarantees (as per https://redis.io/docs/manual/keyspace/):

A full iteration always retrieves all the elements that were present in the collection from the start to the end of a full iteration. This means that if a given element is inside the collection when an iteration is started, and is still there when an iteration terminates, then at some point SCAN returned it to the user.

A full iteration never returns any element that was NOT present in the collection from the start to the end of a full iteration. So if an element was removed before the start of an iteration, and is never added back to the collection for all the time an iteration lasts, SCAN ensures that this element will never be returned.

While playing around with some PoC for cluster wide scan, I realized that the first guarantee can only hold as long as you were connected to the same node during the entire duration of the scanning. If failover occurs, the replica may have a different seed value for the siphash function, so the cursor previously used on the primary would not be in the same place. This is likely less of an issue for SCAN since you normally indicate the node you're talking to, but many CME client transparently handle re-directs for SSCAN/HSCAN/ZSCAN during failovers.

I'm not sure we need to strictly do anything about SCAN, I'm not sure how often SCAN is resumed after a failover. The other commands might though.

I think this is worth addressing, but could be done in three ways:

Simply update the documentation to add the caveat. This doesn't feel right to me, because most users will not really be aware of this.
Add a configuration so that seeds can be set externally. This would allow operators to configure this consistency, but has limited benefit for those that don't know.
Allow replicas to sync their data seed from their primaries. This makes their cursors consistent. We can also persistent this into RDB so that it is still accurate, I'm most in-favor of this.

ref: redis/redis#12440

We never came up with a cohesive answer here

The problem/use-case that the feature addresses

Make it run natively, without cygwin or wsl, on microsoft windows system.

Description of the feature

Currently it does not build or run on windows, make windows users harder to development and test their program using redis protocol. With native windows support, users can setup development environment more easily.
There are some existing work on this, but not actively mantained now.

Alternatives you've considered

• https://github.com/tporadowski/redis/
• https://github.com/microsoftarchive/redis
• cygwin
• wsl
• hyperv / virtualbox with a linux guest os

Additional information

Any additional information that is relevant to the feature request.

Starting a thread to find ideas for folks to propose alternative naming ideas. If you have an idea, please post it here in a single comment. 👍 for ideas that people like. If your name is not descriptive or clear what it means, also please include a description about why.

As per, https://github.com/placeholderkv/placeholderkv/pull/35/files#diff-cccaa466d7e18f8b5fdf9fe8b0d2abd4b29d39f509af8bdb8d3878a0fbf02ac2L11-L12.

The structure of each JSON is somewhat documented in https://redis.io/commands/command-docs/ and https://redis.io/commands/command/

We should probably actually document them for external and internal users.

Many file names, directories, code, docs reference "redis" instead of "valkey". Also, some binary files i.e. valkey-cli, valkey-server have been renamed but some scripts i.e. create-cluster are still referencing redis-cli, redis-server. Probably a good idea to break up into several smaller PR's as the scope is large. Lmk what y'all think, thanks!

It would be great to work on some devendoring of "placeholderkv". Distros (I represent Fedora/EPEL in this context) prefer dynamic linking to system libraries instead of vendored packages with included deps.

It would be great to have native support for using system libs for everything possible in the make file, and perhaps some CI pipelines testing various versions, etc. so that it's easier to monitor if/when things break and on what versions so that expectations can be set.

For our initial compatibility release, I think we want to have a very limited and well defined set of changes. I am proposing we launch a 7.2.4 version, which is clearly compatible with the OSS Redis version 7.2.4. We plan to launch the changes in two tranches RC1 and RC2.

Release candidate 1. 7.2.4-RC1

1. We will build with the following binaries:
   1. valkey-server
   2. valkey-cli
   3. valkey-benchmark
   4. valkey-check-aof
   5. valkey-check-rdb
   6. valkey-sentinel
   7. For compatibility, we also create symlinks to these files with redis-*names in make install (for the installed binaries only), so valkey can be used as a drop-in replacement for Redis.
   8. ☑️ REDIS_FLAGS will be updated to SERVER_FLAGS. Maybe we should also allow REDIS_FLAGS -> SERVER_FLAGS as well, for an extra layer of compatibility. Done in #66.
2. We will introduce 2 new info fields.
   1. valkey_version:7.2.4: This will indicate the valkey_version compatibility.
   2. redis_version:7.2.4: this isn't new, just called out that we will leave it for compatibility.
   3. server_name:valkey: This will provide a way for external clients to determine that this is valkey, if they want.
3. In RDB format, we will add one new aux field and remove one aux field:
   1. valkey-ver:7.2.4: This will indicate that valkey produced the RDB and what version it was generated from. This is only used for compatibility and logging, so it should be safe to add.
   2. redis-ver:7.2.4: This field will be removed. Since it's only used for logging, it should be safe to remove this.
4. Logging and other misc wording.
   1. In order to maximize compatibility, we will leave all logging unchanged, as we expect users may have scripts looking for specific log lines.
   2. We will remove the Redis logo from the startup and also include our website address (when we have one).
5. We need the following LUA compatibility.
   1. We will create a new high level object that maps to the redis object, so that you can do server.call(), etc and other stuff as well as redis.call().
   2. We will update the top level lua version to include the valkey version, server name, while still keeping the redis version for compatibility.
6. Module API compatibility
   1. Continue to fully support REDISMODULE_* for API compatibility
   2. Introduce a second VALKEYMODULE_* API that is find/replace compatible. This will be a secondary namespace, with the intention of all new APIS going here.

RDB represents Redis Database, it is a kind of persistent way for data in redis,
we could change the file name in the future version but it could break the back compatibility.