snapchat / keydb Goto Github PK

A Multithreaded Fork of Redis

License: BSD 3-Clause "New" or "Revised" License

Makefile 0.36% Shell 0.66% C 23.36% C++ 57.90% Ruby 0.34% Tcl 16.80% Smarty 0.01% Assembly 0.10% Dockerfile 0.17% Roff 0.16% Python 0.13%

keydb's People

Contributors

Stargazers

Watchers

Forkers

trucnguyenlam ossfintech shannonman lamlion thaneacheron donfanning joshuajlai mbrukman sandeshbhandari bhanug diegopacheco mdheller linecode shaunstanislauslau raceli eblenkarn jingming11 b-xiang isgasho marceloboeira rahulsoibam zhanglei tomzhang ftt1024 0xflotus hhy5277 wenqzheng githubch flyyi jokerzh suixingbugai stutiredboy yibit frenchair rech mixxim yonggeshidai davidalphafox tide999 lidi100 mark-liang xubingyue newlysoft jangocheng chyuch-github ksnnkyo phpth digimon99 devopsdood gitter-badger soloestoy trevor211 microredis ghz-max jonahharris curtiszimmerman sckevmit iamsingularity batermj asif31iqbal shawvi namles charsyam startime-h qix- emuhedo guoanwu negashev itouba clxdsjyx sadnessofatlantis x77a1 yuguangjie brydzu hulaoan kranthik2 cherrydat btolab billowqiu markcol dd-guo gnk-cloud arunp0 longjohncoder blunney1 eric-weaver nautilus-nico null-exception karthik-cbe segmond hernan604 firejava jashk sd37 prcyangli tchen0123 a-b lisprez baodingfengyun dendisuhubdy

keydb's Issues

keydb crashed again, again

I'm using active-active replication in docker environments

docker -v
Docker version 18.09.6, build 481bc77156
uname -a (CENTOS 7)
Linux ocean-02 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

ERROR MESSAGE:

...........
what(): Operation not permitted
Aborted
terminate called after throwing an instance of 'std::system_error'
what(): Operation not permitted
Aborted
terminate called after throwing an instance of 'std::system_error'
what(): Operation not permitted
Aborted
terminate called after throwing an instance of 'std::system_error'
what(): Operation not permitted
Aborted
terminate called after throwing an instance of 'std::system_error'
what(): Operation not permitted
Aborted
...........

HGETALL returns NIL, Redis returns EMPTY SET

When you run HGETALL against a key that doesnt exist, KeyDB returns a NIL response. Redis returns (empty list or set). This breaks compatibility with some libraries (StackExchange.Redis)

SETUP

docker run -it -p 6391:6379 eqalpha/keydb:latest
docker run -it -p 6390:6379 redis:latest

Test

will@45K9VT2:/mnt/c/Users/wdavis$ redis-cli -p 6390 HGETALL test
(empty list or set)
will@45K9VT2:/mnt/c/Users/wdavis$ redis-cli -p 6391 HGETALL test
(nil)
will@45K9VT2:/mnt/c/Users/wdavis$

Does KeyDB active-active need sentinel?

Does KeyDB require Redis Sentinel? While I can see how they exist here for being compatible with Redis failover, do I need Sentinel when setting up active-active replication?

That is, after the master fails, I can continue writing to the replica with just switching the IP address. Do I need to do anything more?

Error when make

just use make

make[3]: Leaving directory '/home/atn4517/KeyDB-0.9.2/deps/jemalloc'
make[2]: Leaving directory '/home/atn4517/KeyDB-0.9.2/deps'
    CC adlist.o
    CC quicklist.o
    CC ae.o
    CC anet.o
    CC dict.o
    CC server.o
In file included from server.cpp:31:0:
server.h:2331:68: error: declaration of ‘void* calloc(size_t, size_t)’ has a different exception specifier
 void *calloc(size_t count, size_t size) __attribute__ ((deprecated));
                                                                    ^
In file included from server.h:42:0,
                 from server.cpp:31:
/usr/include/stdlib.h:468:14: error: from previous declaration ‘void* calloc(size_t, size_t) throw ()’
 extern void *calloc (size_t __nmemb, size_t __size)
              ^
In file included from server.cpp:31:0:
server.h:2332:49: error: declaration of ‘void free(void*)’ has a different exception specifier
 void free(void *ptr) __attribute__ ((deprecated));
                                                 ^
In file included from server.h:42:0,
                 from server.cpp:31:
/usr/include/stdlib.h:483:13: error: from previous declaration ‘void free(void*) throw ()’
 extern void free (void *__ptr) __THROW;
             ^
In file included from server.cpp:31:0:
server.h:2333:54: error: declaration of ‘void* malloc(size_t)’ has a different exception specifier
 void *malloc(size_t size) __attribute__ ((deprecated));
                                                      ^
In file included from server.h:42:0,
                 from server.cpp:31:
/usr/include/stdlib.h:466:14: error: from previous declaration ‘void* malloc(size_t) throw ()’
 extern void *malloc (size_t __size) __THROW __attribute_malloc__ __wur;
              ^
In file included from server.cpp:31:0:
server.h:2334:66: error: declaration of ‘void* realloc(void*, size_t)’ has a different exception specifier
 void *realloc(void *ptr, size_t size) __attribute__ ((deprecated));
                                                                  ^
In file included from server.h:42:0,
                 from server.cpp:31:
/usr/include/stdlib.h:480:14: error: from previous declaration ‘void* realloc(void*, size_t) throw ()’
 extern void *realloc (void *__ptr, size_t __size)
              ^
Makefile:278: recipe for target 'server.o' failed
make[1]: *** [server.o] Error 1
make[1]: Leaving directory '/home/user/KeyDB-0.9.2/src'
Makefile:6: recipe for target 'all' failed
make: *** [all] Error 2

ARM Support

Arm currently compiles, but crashes with multiple threads. Our fastlock implementation assumes X86's very strong memory model. This needs to be updated for ARM.

Versioning

I'm trying to understand the logic behind current versioning you use (e.g. 0.9.4 to 0.9.4-r2, instead of 0.9.5). I'm an Linux rpm packager and this type of versioning creates issues on the long term.

Regards,

Floren

MALLOC =memkind compile related issues

I created successfully a rpm with MALLOC=jemalloc and jemalloc-devel system libraries, on CentOS 7.6 and GCC 5.3.1. Everything fails when I try to use either deps/memkind, or memkind-devel-1.7.0 system libraries.

%global modules_abi     1
%global source_name     KeyDB

Name:                   keydb
Version:                0.9.4
Release:                1%{?dist}
Summary:                Multithreaded fork of Redis
License:                BSD
URL:                    https://github.com/JohnSully/KeyDB
Source0:                %{url}/archive/v%{version}.tar.gz#/%{source_name}-%{version}.tar.gz
Source1:                %{name}.logrotate
Source2:                %{name}.service
Source3:                %{name}-sentinel.service
Source4:                %{name}.limit
Source5:                %{name}.tmpfiles
Source6:                %{name}.shutdown
Patch:                  0002-install-redis-check-rdb-as-a-symlink-instead-of-dupl.patch
ExclusiveArch:          x86_64
BuildRequires:          gcc >= 5.3.0
BuildRequires:          jemalloc-devel
BuildRequires:          libuuid-devel
BuildRequires:          memkind-devel
BuildRequires:          nasm
BuildRequires:          redis-rpm-macros
BuildRequires:          systemd
BuildRequires:          tcl-devel
Requires:               gawk
Requires:               logrotate
Provides:               bundled(hiredis)
Provides:               bundled(linenoise)
Provides:               bundled(lua-libs)
Provides:               bundled(lzf)
Provides:               %{name}(modules_abi)%{?_isa} = %{modules_abi}
Conflicts:              redis
%{?%systemd_requires}


%prep
%autosetup -n %{source_name}-%{version} -p1
%{__rm} -rf deps/jemalloc
%{__rm} -rf deps/memkind
%{__sed} -i -e '/cd jemalloc && /d' deps/Makefile
%{__sed} -i -e '/cd memkind && /d' deps/Makefile
%{__sed} -i -e 's|../deps/jemalloc/lib/libjemalloc.a|-ljemalloc -ldl|g' src/Makefile
%{__sed} -i -e 's|-I../deps/jemalloc.*|-DJEMALLOC_NO_DEMANGLE -I%{_includedir}/jemalloc|g' src/Makefile
%{__sed} -i -e 's|../deps/memkind/src/.libs/libmemkind.a|-lmemkind|g' src/Makefile
%{__sed} -i -e 's|-I../deps/memkind.*|-I%{_includedir}/memkind|g' src/Makefile
%{__sed} -i -e 's|redis|%{name}|g' redis.conf
%{__sed} -i -e 's|%{name}\.io|redis.io|g' redis.conf
%{__sed} -i -e 's|^logfile .*$|logfile %{_localstatedir}/log/%{name}/%{name}.log|g' redis.conf
%{__sed} -i -e '$ alogfile %{_localstatedir}/log/%{name}/sentinel.log' sentinel.conf
%{__sed} -i -e 's|^dir .*$|dir %{_sharedstatedir}/%{name}|g' redis.conf
api=$(sed -n -e 's/#define REDISMODULE_APIVER_[0-9][0-9]* //p' src/redismodule.h)
if [ "$api" != "%{modules_abi}" ]; then
    : Error: Upstream API version is now ${api}, expecting %%{modules_abi}.
    : Update the modules_abi macro, the rpmmacros file, and rebuild.
    exit 1
fi


%build
%make_build CFLAGS+="%{optflags} -fPIC" LDFLAGS="%{__global_ldflags}" \
    MALLOC=memkind PREFIX=%{buildroot}%{_prefix} all

    CC storage.o
storage.cpp: In function 'int forkFile()':
storage.cpp:124:35: error: 'memkind_tmpfile' was not declared in this scope
     memkind_tmpfile(PMEM_DIR, &fdT);
                                   ^
storage.cpp:125:46: error: 'memkind_fd' was not declared in this scope
     if (ioctl(fdT, FICLONE, memkind_fd(mkdisk)) == -1)
                                              ^
storage.cpp: In function 'void storage_init(const char*, size_t)':
storage.cpp:177:44: error: 'memkind_fd' was not declared in this scope
         posix_fallocate64(memkind_fd(mkdisk), 0, cbFileReserve);
                                            ^
storage.cpp: In function 'size_t salloc_usable_size(void*)':
storage.cpp:214:60: error: 'memkind_malloc_usable_size' was not declared in this scope
     return memkind_malloc_usable_size(kindFromPtr(ptr), ptr);
                                                            ^
storage.cpp: In function 'void handle_postfork_child()':
storage.cpp:269:39: error: 'memkind_fd' was not declared in this scope
     int fdOriginal = memkind_fd(mkdisk);
                                       ^
storage.cpp:270:39: error: 'memkind_pmem_remapfd' was not declared in this scope
     memkind_pmem_remapfd(mkdisk, fdNew);
                                       ^
make[1]: *** [storage.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory `/builddir/build/BUILD/KeyDB-0.9.4/src'
make: *** [all] Error 2

I'm only interested to build using system libraries. I tried building with the /deps but it failed also.

A question about KEYS variable in lua scripts

As I understand it, in mainline redis the difference between KEYS and ARGV is that Redis will make sure that the KEYS values exist on the current slot in a clustered environment and will otherwise fail with CROSSSLOT if one of them does not.

This limits the ability to write scripts that perform dynamic key composition (where a key is built up in lua-land on the instance) in a clustered environment, making the two features mutually exclusive - which is unfortunate.

However, with KeyDB the clustering/replication properties have changed a bit. Is the KEYS/dynamic key generation limitation still applicable in KeyDB? Or can I safely use dynamic keys in a clustered environment?

I understand that KeyDB provides both KEYS and ARGV for compatibility but I'm interested in whether or not dynamic key referencing in a clustered environment will work as intended or if it still follows the mainline Redis limitations.

Thanks John! :)

EDIT: I just read through most of the HN thread and realize now that I wouldn't even want a cluster mode with KeyDB (thanks!). I'm still interested in the answer, purely for academic purposes :)

reserved identifier violation

I would like to point out that identifiers like “__AE_H__” and “__RDB_H” do not fit to the expected naming convention of the C++ language standard.
Would you like to adjust your selection for unique names?

zrangebyscore return (nil)

simple redis (redis:alpine)

redis-cli
127.0.0.1:6379> ZADD myzset 1 "one"
(integer) 1
127.0.0.1:6379> ZADD myzset 2 "two"
(integer) 1
127.0.0.1:6379> ZADD myzset 3 "three"
(integer) 1
127.0.0.1:6379> ZRANGEBYSCORE myzset -inf +inf
1) "one"
2) "two"
3) "three"
127.0.0.1:6379> ZRANGEBYSCORE myzset (1 (2
(empty list or set)

simple keydb (eqalpha/keydb:x86_64_v0.9.5)

keydb-cli -a password
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> ZADD myzset 1 "one"
(integer) 1
127.0.0.1:6379> ZADD myzset 2 "two"
(integer) 1
127.0.0.1:6379> ZADD myzset 3 "three"
(integer) 1
127.0.0.1:6379> ZRANGEBYSCORE myzset -inf +inf
1) "one"
2) "two"
3) "three"
127.0.0.1:6379> ZRANGEBYSCORE myzset (1 (2
(nil)

How to configure a cluster with several KeyDBs ?

I can't find documentation about use KeyDB as a cluster

v0.9.6 tag syntax

The 0.9.6 tag dropped the 'v' prefix (and is thus harder to notice in the list) than v0.9.5, v0.9.4, etc. Looks accidental?

Replication information not available, can't generate the RDB file right now

We are running a two master setup in active-active mode, both instances crashed a few hours ago after running fine for about a week and now I am trying to get them back in sync. I started one of the master's and then proceeded to start the second master after the AOF file from the first master was loaded (I did remove the AOF file from the second master in order to get a clean state). After several attempts with the second master to sync with the first master I am now getting this error: (which loops indefinitely)

1:S 14 Jul 2019 19:52:41.104 * MASTER <-> REPLICA sync started
1:S 14 Jul 2019 19:52:41.104 * Non blocking connect for SYNC fired the event.
1:S 14 Jul 2019 19:52:41.105 * Master replied to PING, replication can continue...
1:S 14 Jul 2019 19:52:41.106 * Partial resynchronization not possible (no cached master)
1:S 14 Jul 2019 19:52:41.106 * Master does not support PSYNC or is in error state (reply: -ERR BGSAVE failed, replication can't continue)
1:S 14 Jul 2019 19:52:41.106 * Retrying with SYNC...
1:S 14 Jul 2019 19:52:41.106 # I/O error reading bulk count from MASTER: Resource temporarily unavailable
1:S 14 Jul 2019 19:52:41.662 * Replica XX.XX.XX.XXX:6380 asks for synchronization
1:S 14 Jul 2019 19:52:41.662 * Full resync requested by replica XX.XX.XX.XXX:6380
1:S 14 Jul 2019 19:52:41.662 * Starting BGSAVE for SYNC with target: disk
1:S 14 Jul 2019 19:52:41.662 # BGSAVE for replication: replication information not available, can't generate the RDB file right now. Try later.
1:S 14 Jul 2019 19:52:41.662 # BGSAVE for replication failed
...
BGSAVE for replication: replication information not available, can't generate the RDB file right now. Try later.

What is the preferred approach to recovering from a crash in an active-active replication setup?

Completion of error handling

Would you like to add more error handling for return values from functions like the following?

pthread_cond_init ⇒ bioInit
pthread_mutex_init ⇒ initServerConfig

Release 5 sieze-up, followed by crash on replica promotion

We saw a KeyDB server built from the RELEASE_5 branch become completely unresponsive yesterday. Our systems responded by promoting a replica which crashed immediately when receiving SLAVEOF NO ONE

Unfortunately I don't have many details about the master to share, other than noting that the overall system was showing nominal behavior and load, with log output stopping around 12:19 UTC. The replica logs below start at about that time when the master connection is dropped, and our system attempts a promotion at 12:24 UTC.

Jul 14 12:19:55 keydb-server[2905]: 2905:S 14 Jul 2019 12:19:55.834 # MASTER timeout: no data nor PING received...
Jul 14 12:19:55 keydb-server[2905]: 2905:S 14 Jul 2019 12:19:55.834 # Connection with master lost.
Jul 14 12:19:55 keydb-server[2905]: 2905:S 14 Jul 2019 12:19:55.834 * Caching the disconnected master state.
Jul 14 12:19:55 keydb-server[2905]: 2905:S 14 Jul 2019 12:19:55.834 * Connecting to MASTER xxxxx
Jul 14 12:19:55 keydb-server[2905]: 2905:S 14 Jul 2019 12:19:55.836 * MASTER <-> REPLICA sync started
Jul 14 12:22:05 keydb-server[2905]: 2905:S 14 Jul 2019 12:22:05.349 # Error condition on socket for SYNC: Connection timed out
Jul 14 12:22:06 keydb-server[2905]: 2905:S 14 Jul 2019 12:22:06.073 * Connecting to MASTER xxxxx
Jul 14 12:22:06 keydb-server[2905]: 2905:S 14 Jul 2019 12:22:06.077 * MASTER <-> REPLICA sync started
Jul 14 12:24:16 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:16.421 # Error condition on socket for SYNC: Connection timed out
Jul 14 12:24:17 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:17.316 * Connecting to MASTER xxxxx
Jul 14 12:24:17 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:17.320 * MASTER <-> REPLICA sync started
Jul 14 12:24:48 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:48.290 # Setting secondary replication ID to 9b8fd931f161c8be497aa79b23823c188394905d, valid up to offset: 4659745187. New replication ID is 78b24b65287543c6c19c9e3ba964996d2881558c
Jul 14 12:24:48 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:48.290 * Discarding previously cached master state.
Jul 14 12:24:48 keydb-server[2905]: AE_ASSER FAILURE
Jul 14 12:24:48 keydb-server[2905]: === KEYDB BUG REPORT START: Cut & paste starting from here ===
Jul 14 12:24:48 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:48.290 # KeyDB 5.0.0 crashed by signal: 11
Jul 14 12:24:48 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:48.290 # Crashed running the instruction at: 0x432eee
Jul 14 12:24:48 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:48.290 # Accessing address: (nil)
Jul 14 12:24:48 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:48.290 # Failed assertion: <no assertion failed> (<no file>:0)
Jul 14 12:24:48 keydb-server[2905]: ------ STACK TRACE ------
Jul 14 12:24:48 keydb-server[2905]: EIP:
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(aeDeleteFileEvent+0x3e) [0x432eee]
Jul 14 12:24:48 keydb-server[2905]: Backtrace:
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(logStackTrace(ucontext_t*)+0x39) [0x4851a9]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(sigsegvHandler(int, siginfo_t*, void*)+0xa4) [0x485824]
Jul 14 12:24:48 keydb-server[2905]: /lib64/libpthread.so.0(+0x121f0) [0x7f964976c1f0]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(aeDeleteFileEvent+0x3e) [0x432eee]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(undoConnectWithMaster(redisMaster*)+0x26) [0x457596]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(cancelReplicationHandshake(redisMaster*)+0x25) [0x457635]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(replicationUnsetMaster(redisMaster*)+0x97) [0x457ef7]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(replicaofCommand(client*)+0x17c) [0x45886c]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(call(client*, int)+0xbd) [0x43a6dd]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(execCommand(client*)+0xc7) [0x481ff7]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(call(client*, int)+0xbd) [0x43a6dd]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(processCommand(client*, int)+0x789) [0x43ebc9]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(processCommandAndResetClient(client*, int)+0x21) [0x447391]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(processInputBuffer(client*, int)+0x7a) [0x44bbba]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(readQueryFromClient(aeEventLoop*, int, void*, int)+0x338) [0x44c138]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(ProcessEventCore+0xe9) [0x433409]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(aeProcessEvents+0x14d) [0x43369d]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(aeMain+0x2d) [0x433a0d]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(workerThreadMain(void*)+0x90) [0x437aa0]
Jul 14 12:24:48 keydb-server[2905]: /lib64/libpthread.so.0(+0x754b) [0x7f964976154b]
Jul 14 12:24:48 keydb-server[2905]: /lib64/libc.so.6(clone+0x3f) [0x7f96494992ff]

Client details for the replica:

Jul 14 12:24:48 keydb-server[2905]: ------ CLIENT LIST OUTPUT ------
Jul 14 12:24:48 keydb-server[2905]: id=13 addr=172.16.101.10:44516 fd=98 name=redisgreen-monitoring age=197244 idle=24 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=slowlog
Jul 14 12:24:48 keydb-server[2905]: id=16 addr=172.16.101.11:57666 fd=96 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
Jul 14 12:24:48 keydb-server[2905]: id=17 addr=172.16.101.11:57668 fd=97 name=redisgreen age=0 idle=0 flags=x db=0 sub=0 psub=0 multi=2 qbuf=123 qbuf-free=32645 obl=27 oll=0 omem=0 events=r cmd=exec
Jul 14 12:24:48 keydb-server[2905]: ------ CURRENT CLIENT INFO ------
Jul 14 12:24:48 keydb-server[2905]: id=17 addr=172.16.101.11:57668 fd=97 name=redisgreen age=0 idle=0 flags=x db=0 sub=0 psub=0 multi=2 qbuf=123 qbuf-free=32645 obl=27 oll=0 omem=0 events=r cmd=exec
Jul 14 12:24:48 keydb-server[2905]: argv[0]: 'slaveof'
Jul 14 12:24:48 keydb-server[2905]: argv[1]: 'no'
Jul 14 12:24:48 keydb-server[2905]: argv[2]: 'one'

Is this RESP protocol-compatible with mainline Redis?

And if it is, perhaps a note mentioning that in the readme would serve beneficial.

CRDT based synchronization strategy - feature

This ticket is to track the feature request.

Enable CRDT based synchronization strategy feature in keysdb.

Starting one client causes another to crash with latest build

https://gist.github.com/PurpleIsEverything/c22b4bb4242c887d5d796dce8500e55e

Failed Tests

I am using Ubuntu 18.04 LTS and running make tests throws the followings:

!!! WARNING The following tests failed:
*** [err]: Server is able to generate a stack trace on selected systems in tests/integration/logging.tcl
expected stack trace not found into log file
*** [err]: PSYNC2: cluster is consistent after load (x = 52143) in tests/integration/psync2.tcl
Instance #1 x variable is inconsistent

Should I be concerned?

Is it possible to use that software to build your own DBaaS solution or license it on-prem basis?

RREPLAY Command doesn't execute with permissions of the client

The root cause is the fake user we create to execute commands in RREPLAY doesn't have the authentication information of the client executing the RREPLAY command. This may cause commands to fail due to permission issues.

Add test for replicaof config

This runs the createMaster function before databases are allocated and has been broken more than once. Current tests only create replicas via command after boot has completed and so do not validate this scenario.

The test infra needs to be modified slightly to support this.

The latest example of this is issue #58

Mixing multiple masters and Active Replication causes reconnection loop

This issue tracks the following two issues:

Reconnection loops
Command loops

Its caused by the feedback loop enabled by Active Replication. These issues are resolved for single masters but reappear with multiple masters and more complex topologies.

KeyDB compile error (CentOS)

ENV:
CentOS 6.5 x86_64 with devtoolset-3-gcc (4.9.2)
I use
make CFLAGS="-march=haswell -m128bit-long-double -m64 " MALLOC=jemalloc
I get this info:
server.cpp: In function 'char* genRedisInfoString(const char*)':
server.cpp:4453:29: error: expected ')' before 'PRIu64'
"long_lock_waits:%" PRIu64 "\r\n",
^
server.cpp:4459:36: warning: spurious trailing '%' in format [-Wformat=]
fastlock_getlongwaitcount());
^
server.cpp:4459:36: warning: too many arguments for format [-Wformat-extra-args]
make[1]: *** [server.o] Error 1
make[1]: Leaving directory `/opt/opstoolset/source/redis/src/KeyDB-0.9.4-r2/src'

How to fix this ? very very thanks :-)

some clarifications regarding replication

Hello,

I need few clarifications regarding the active-active replication and the performance impact.

The replication is async ?

For example I have 2 datacenters, in each datacenter for each shard I'm planning to add 2 nodes in active- active replication.

Each shard in DC1 will have an associated shard in DC2 as well with the same number of nodes.

So in the end each redis node from a shard will have 3 replicas (1 in the same dc and 2 in the other dc).

In case the replication is sync the performance impact will be high so this is my biggest concern in this moment comparing with
the https://github.com/Netflix/dynomite where the replication is async.

How fail-over works ?

Let's suppose one node goes down. While is down the clients will send the read/writes to the other replicas.
After some time the node is fixed and started back. The node will accept connections from clients only after it's in sync with the other replicas ?

Because otherwise the clients can start connecting to it and send requests to keys that were not already propagated from replicas

Kind regards,
Silviu

Section addressing non-serialiazability of a multithreaded redis

A significant reason is great is that scripts and MULTI-EXEC commands are isolated from other clients making it easy to reason about transactions. This comes from the single threaded nature of redis. I didn't find a section in your README addressing the concern that KeyDB would allow 2 transactions to run concurrently.

Could not connect to Redis when choosing a non-default port

Upon configuring KeyDB to use a non-default port the following error occurs:

$ keydb-cli
Could not connect to Redis at 127.0.0.1:6379: Connection refused
not connected>

In this instance I am using 6366 but it tries to connect to 6379 as shown above.
I've set the port to 6366 when running install_sever.sh and the config file contains the following:
port 6366
KeyDB is running in the background just fine and my website happily uses it through port 6366.

OS: Ubuntu 16.04

Setup CI Servers

Lots of build errors on specific platforms. This tracks CI on Debian, Ubuntu, and Fedora.

Remove ifdef hack for deprecated malloc+family

Some users are reporting that these deprecated prototypes fail to compile because the exception specifier doesn't match. The hack removes the deprecated flags for C++ builds (C does not have exception specifiers).

These should still flag an error in C++ so this issue tracks reenabled the deprecated flag for C++.

Hack checked in here: a9e525c

Unexpected reply to PSYNC from master: -NOAUTH

I am trying to add a replica slave from host2:6380 to my master on host1:6379 and added
masterauth into the appropriate keydb-slave.conf.
The master is running with an identical requirepass definition.
However as the slave tries to connect to master it fails with:
Unexpected reply to PSYNC from master: -NOAUTH Authentication required.

Would someone know what I am doing wrong? Thx

Segfault on databases > 16

Thank you for your work on KeyDB!

This problem is reproducible on OSX and Linux - configure with databases 17 (or any number higher than 16) and keydb-server will segfault on startup with (trace from Amazon Linux 2):

=== KEYDB BUG REPORT START: Cut & paste starting from here ===
2841:M 12 Jul 2019 05:51:37.727 # KeyDB 5.0.0 crashed by signal: 11
2841:M 12 Jul 2019 05:51:37.727 # Crashed running the instruction at: 0x43b081
2841:M 12 Jul 2019 05:51:37.727 # Accessing address: 0x48
2841:M 12 Jul 2019 05:51:37.727 # Failed assertion: <no assertion failed> (<no file>:0)

------ STACK TRACE ------
EIP:
./keydb-server *:6379(serverCron(aeEventLoop*, long long, void*)+0x1a1) [0x43b081]

Backtrace:
./keydb-server *:6379(logStackTrace(ucontext_t*)+0x39) [0x4851a9]
./keydb-server *:6379(sigsegvHandler(int, siginfo_t*, void*)+0xa4) [0x485824]
/lib64/libpthread.so.0(+0x121f0) [0x7fe1f24981f0]
./keydb-server *:6379(serverCron(aeEventLoop*, long long, void*)+0x1a1) [0x43b081]
./keydb-server *:6379(aeProcessEvents+0x223) [0x433773]
./keydb-server *:6379(aeMain+0x2d) [0x433a0d]
./keydb-server *:6379(workerThreadMain(void*)+0x90) [0x437aa0]
/lib64/libpthread.so.0(+0x754b) [0x7fe1f248d54b]
/lib64/libc.so.6(clone+0x3f) [0x7fe1f21c52ff]

------ INFO OUTPUT ------
Segmentation fault

Active Replicas Report Themselves as Slaves in INFO command

This is just an informational bug in the output of the command.

Feature Request, Expire members of a set

Hi,

One of the much requested features, to expire individual members of a set, is unlikely to get implemented in Redis, hoping KeyDb can take to improve it

redis/redis#135

multi-master installation steps.

Hello,

We are trying out the evaluation for multi master setup where keydb would be installed in 2 different zones having multi master mode enabled.

I went through this link provided in the wiki https://github.com/JohnSully/KeyDB/wiki/Multimaster-Support but i was unable to figure out the complete configurations required.

Only parameter which i found is multi-master yes in the configuration, i am not sure how would i configure other master to which the sync would happen.

Could you please help me on this.

Thanks.
Shiva

KeyDB reports very low redis_version

Hello!

First I'd like to thank you for all your hard work. I've noticed that via the INFO command, KeyDB is reporting a very low redis_version. This unfortunately breaks every piece of software which tests against this value.

Is it possible that instead of yielding the version of KeyDB in that field, it'd be better for compatibility purposes, to yield a version of Redis that the build is compatible with and maintain version information for KeyDB in a separate key like keydb_version?

Getting timeout when adding document to Redisearch

Hi,
I'm trying to use redisearch with keydb. Afaik, keydb supports modules. I can create indices but I get timeout when adding document. You can reproduce this issue with these codes:

root@keydb-0:/data# keydb-cli
127.0.0.1:6379> FT.CREATE idx SCHEMA id NUMERIC SORTABLE name TEXT SORTABLE
OK
127.0.0.1:6379> FT.ADD idx 1 1.0 FIELDS id 1 name 'NameTest'

[Q] keys scan command

REDIS warning..
..........................................
When the number of keys in redis (use DBSIZE to know the count) is less, this method can be used but when redis size grows, using keys cmd is extremely dangerous for performance and highly discouraged in production environment since redis is single threaded, keys cmd will block all other operations on redis and all the clients will have to wait for its completion.
..........................................

Is KeyDB subject to the same constraints or not ?

Build Errors

I hope I am not wasting your time, because I intend not to.
This is the results of the Redis tests on my Ubuntu 16.04 LTS:

!!! WARNING The following tests failed:

*** [err]: Command Filter REDISMODULE_CMDFILTER_NOSELF works as expected in tests/modules/commandfilter.tcl
Expected '' to be equal to '{set mykey @log}'
*** [err]: SLAVE can reload "lua" AUX RDB fields of duplicated scripts in tests/integration/replication-3.tcl
Replication not started.

If this is the expected result, please ignore this and close the thread.

build error on centos 7

keydb_v0.9.5, i am getting error with centos 7

cc: error: unrecognized command line option ‘-std=c++14’
make[1]: *** [ae.o] Error 1
make[1]: Leaving directory `/home/hadn/dynomite/KeyDB-0.9.5/src'
make: *** [all] Error 2

[hadn@vsftpd src]$ uname -a
Linux vsftpd 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Memory accumulates when running long lua scripts

The following script repeatedly sets a key to different values. Memory grows quickly and excessively as the script is run (over many GB). If the script is able to run to completion the memory is freed.

Script:
for i = 1, 10000000000, 1 do
redis.call("SET", "ZzZ_MYKEYZzZ"..i.."key", i)
end

Repro:
./keydb-cli --eval /tmp/bench.lua

This behavior is particularly bad when FLASH backing is used as it leads to write amplification.

crash from failed assertion in networking.cpp

Hi there, in case it's useful here's a crash from a KeyDB instance built off the RELEASE_5 branch last week, using 6 threads, virtual network interface has 8 queues. I can provide more details and/or sanitize the INFO/CLIENT payloads if it'd be helpful.

Jul 14 03:20:04 keydb-server[2881]: === KEYDB BUG REPORT START: Cut & paste starting from here ===
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # === ASSERTION FAILED CLIENT CONTEXT ===
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # client->flags = 2097152
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # client->fd = 425
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # client->argc = 2
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # client->argv[0] = ">" (refcount: 2)
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # client->argv[1] = "queue:keydb2" (refcount: 1)
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # === ASSERTION FAILED ===
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # ==> networking.cpp:2633 'c->cmd != NULL' is not true
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # (forcing SIGSEGV to print the bug report.)
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # KeyDB 5.0.0 crashed by signal: 11
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # Crashed running the instruction at: 0x482738
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # Accessing address: 0xffffffffffffffff
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # Failed assertion: c->cmd != NULL (networking.cpp:2633)
Jul 14 03:20:04 keydb-server[2881]: ------ STACK TRACE ------
Jul 14 03:20:04 keydb-server[2881]: EIP:
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(_serverAssert+0x78) [0x482738]
Jul 14 03:20:04 keydb-server[2881]: Backtrace:
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(logStackTrace(ucontext_t*)+0x39) [0x4851a9]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(sigsegvHandler(int, siginfo_t*, void*)+0xa4) [0x485824]
Jul 14 03:20:04 keydb-server[2881]: /lib64/libpthread.so.0(+0x121f0) [0x7fc293da61f0]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(_serverAssert+0x78) [0x482738]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(rewriteClientCommandVector(client*, int, ...)+0x120) [0x447df0]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(blockingPopGenericCommand(client*, int)+0x1db) [0x4666bb]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(call(client*, int)+0xbd) [0x43a6dd]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(processCommand(client*, int)+0x789) [0x43ebc9]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(processCommandAndResetClient(client*, int)+0x21) [0x447391]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(processInputBuffer(client*, int)+0x7a) [0x44bbba]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(readQueryFromClient(aeEventLoop*, int, void*, int)+0x338) [0x44c138]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(ProcessEventCore+0xe9) [0x433409]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(aeProcessEvents+0x14d) [0x43369d]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(aeMain+0x2d) [0x433a0d]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(workerThreadMain(void*)+0x90) [0x437aa0]
Jul 14 03:20:04 keydb-server[2881]: /lib64/libpthread.so.0(+0x754b) [0x7fc293d9b54b]
Jul 14 03:20:04 keydb-server[2881]: /lib64/libc.so.6(clone+0x3f) [0x7fc293ad32ff]

Add Docker link to Readme

Today docker is vastly used so I really recommend updating project README with link to the docker image at https://hub.docker.com/r/eqalpha/keydb

CURP - fast replication nearly for free and without the current caveats

There is a "new" approach to very fast replication called CURP (Consistent Unordered Replication Protocol).

It shall not have the caveats the current replication scheme in Redis has and shall not be difficult to implement (actually it was already partially implemented for Redis to test it - don't know though where the source lies and what is the license).

Would you consider implementing such low hanging fruit?

I've made a similar ticket directly for Redis, but as Redis is under the hood of a company with specific opinions, I'm opening it here as well as I'd like Redis-like DBs thrive.

More test results

Ubuntu 18.04 LTS - Kernel v5.1.9

Providing only the relevant part:

[34/52 done]: unit/type/list-3 (48 seconds)
Testing unit/wait
[err]: FLUSHDB ASYNC can reclaim memory in background in tests/unit/lazyfree.tcl
Expected condition '$peak_mem > $orig_mem+1000000' to be true (13838096 > 13734512+1000000)