Use LuaJIT2 instead of original Lua interpreter about redis HOT 18 CLOSED

Hello,

@Utros: about Lua JIT, sure since it is a drop in replacement I've a long term plan of testing it if we'll incur into speed issues with the current interpreter. It is a switch that we can always do and I'm taking this in mind. However as long as there is no evidence that it is a big win for us (probably not, as we never execute algorithmic-style code in scripts) otherwise having the standard Lua interpreter that compiles fast and clean with an "ANSI" target is better.

@jbergstoem: we'll never support linking against a different version of Lua, as from our point of view scripting is a feature of Redis, so you can't exclude scripting nor you can modify it. A redis version 2.6.5 (for instance) will be able to run a set of scripts, in all the supported architectures, without any differences from a computer to another. So we consider the behavior and functions available in a specific Lua interpreter as our feature and specification. For the same reason we'll probably deny the possibility of loading external code into the interpreter.

Thanks, closing the issue for now since it is a long term business ;)

from redis.

While at it, I would like to see something in style with "USE_LUA_EXTERNAL" instead of statically linking to a bundled version of lua. This would make it easier to switch lua implementation as well.

from redis.

@jbergstroem AFAIK, external dependences isn't Redis-way.

And Redis also uses a bit modified version of Lua (check out comits), so you'll need to replace some functions in your dinamicaly linked library by your own.

from redis.

@Utros: It would sure be a nice way to handle the feature requested in this item. Lets not hijack this thread, but do you know if there's more information about why using shared libraries isn't the "Redis way"?

from redis.

@jbergstroem It doesn't matter how do you link the library, but as i see, redis is intended to be small, fast and have no external dependences (only computer, supported OS and c compiler).

It is just my point of view and redis developers may think different way.

from redis.

I made of fork of Redis which uses the LuaJIT VM instead of the vanilla ("PUC-Lua") VM. You can find it here:
https://github.com/neomantra/redis/tree/luajit

It passes all the tests, but doesn't include the libraries that Redis bundles (cjson, struct, and cmsgpack). In simple benchmarks, it does not perform better than the Lua VM: https://github.com/neomantra/redis/wiki/LuaJIT-Performance

I'm working on moving more of the implementation from C to Lua which does help some (~15% in those tests)... I'll post more when it is ready (if ever). My personal goal is to store C structures in Redis and then manipulate them in-server using the LuaJIT FFI. This is fraught with security/stability issues (e.g. you can very easily deref a null pointer and crash the server), so I don't expect any of this to ever converge with mainline Redis. But it's a fun experiment!

from redis.

Here is my use case for this. I use redis for cache and store invalidation information using redis sets. So I need to update both cache and invalidation information atomically. I used transactions before, but moved to lua scripts, which made my code both simpler and faster. However, now I have some non-trivial lua scripts and the majoriy of time they take are not spent in redis calls.

from redis.

Hey, I want to be honest: currently it is very unlikely we move away from the standard Lua interpreter, but moreover without benchmarks the ability of this idea to move forward is near-zero... To propose a speed related change without good quality benchmarks is mostly pointless.

from redis.

p.s and @neomantra initial attempts at testing it shows no performance gains for simple scripts. Another evidence of problems with this approach. Btw also make sure to test stuff with latest Redis since Lua scripting execution script was improved. It is mostly dominated by cache misses and alloc/dealloc patterns.

from redis.

I understand it's unlikely. Just wanted to show that there are some interest out there. I'll return with benchmarks once I get to that.

from redis.

@Suor You can do an easy first pass of this by installing LuaJIT on your system using LD_PRELOAD since the LuaJIT .so should be drop-in compatible with the Lua 5.1 .so.

Beyond that, my (abandoned) branch tried to make the Redis implementation spend more time in Lua and leverage the FFI to access Redis structures; in particular I wanted long LuaJIT traces (i.e. no interpreter) and zero marshalling. I also did work to make it easier to have a Lua sandbox (e.g. like to specify the loading of the FFI library, which was my impetus for trying this at all).

If you are doing this for performance, also try LuaJIT's 2.1 branch. I think that some of its new features (e.g. trace stitching) might improve the situation here. But it is alpha and people are still reporting bugs.

from redis.

Also of note: FreeBSD provides an option to automatically patch in support for LuaJIT instead of regular Lua when installing from ports: http://svnweb.freebsd.org/ports/head/databases/redis/Makefile?view=markup&pathrev=357277

For use on your own systems, it's pretty easy to replace Lua with LuaJIT inside Redis since LuaJIT is based on Lua 5.1 and Redis uses regular Lua 5.1.

from redis.

For what it's worth I've put together a very minimal fork of the 5.0.3 sources, embedded LuaJIT 2.0.5 (latest stable). All tests are passing. https://github.com/coleifer/redis-luajit

from redis.

@coleifer, have you been using the luajit fork in production? and if you have, do you notice any differences?

from redis.

No, not in production, just for personal/hobby projects. It works fine, but I have not bothered to profile it, nor do I have any measurements to share.

from redis.

@coleifer: I did some quick and dirty tests.

This is about 30x quicker on luajit.

EVAL "return (function ()\n  local a = 0\n  while a < 1000000000 do\n    a = (a + 1)\n  end\n  return a\nend)()" 0

Though calling into redis - ie GET and SET is roughly the same.

Would it be possible for you to do an update to the newest 5.x version with LuaJIT-2.1.0-beta3?

I did try to compile the newest luajit on your repo (just changing the versions on the Makefile) but am having issues compiling. I think I'm missing something.

/usr/bin/ld: scripting.o: in function `luaLoadLibraries':
/root/dev/buffer/redis-luajit/src/scripting.c:851: undefined reference to `luaopen_cjson'
/usr/bin/ld: /root/dev/buffer/redis-luajit/src/scripting.c:852: undefined reference to `luaopen_struct'
/usr/bin/ld: /root/dev/buffer/redis-luajit/src/scripting.c:853: undefined reference to `luaopen_cmsgpack'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:219: redis-server] Error 1
make[1]: Leaving directory '/root/dev/buffer/redis-luajit/src'

from redis.

I figured out why those errors were occurring.

I've reconfigured the 6.2 and 5.0 branches to use the newest luajit v2.1 branch:

https://github.com/zcaudate/redis-luajit/tree/6.2-luajit
https://github.com/zcaudate/redis-luajit/tree/5.0-luajit

from redis.

GitHub workflow has also been setup.

All 5.0 Tests are passing on x86_64 except one (msgpack related):

• https://github.com/zcaudate/redis-luajit/runs/4647189983?check_suite_focus=true

*** [err]: EVAL - cmsgpack can pack and unpack circular references? in tests/unit/scripting.tcl
1566
Expected '82a17905a17881a17882a17905a17881a17882a17905a17881a17882a17905a17881a17882a17905a17881a17882a17905a17881a17882a17905a17881a17882a17905a17881a178c0 1 1' to equal or match '82a17881a17882a17881a17882a17881a17882a17881a17882a17881a17882a17881a17882a17881a17882a17881a178c0a17905a17905a17905a17905a17905a17905a17905a17905 1 1'

All 6.2 Tests are passing until [62/64] (Not sure but I think they are experimental)

https://github.com/zcaudate/redis-luajit/runs/4647186532?check_suite_focus=true

from redis.