Giter VIP home page Giter VIP logo

tarantool / tarantool Goto Github PK

View Code? Open in Web Editor NEW
3.4K 156.0 378.0 109.83 MB

Get your data in RAM. Get compute close to data. Enjoy the performance.

Home Page: https://www.tarantool.io

License: Other

Shell 0.23% CMake 1.06% C++ 8.00% C 33.98% Lua 55.50% Python 0.84% Makefile 0.06% GDB 0.01% Ragel 0.03% Yacc 0.24% Rich Text Format 0.01% HCL 0.01% Perl 0.03% Dockerfile 0.01%
tarantool database transactions in-memory disk lua msgpack appserver

tarantool's Introduction

Tarantool

Actions Status Code Coverage OSS Fuzz Telegram GitHub Discussions Stack Overflow

Tarantool is an in-memory computing platform consisting of a database and an application server.

It is distributed under BSD 2-Clause terms.

Key features of the application server:

Key features of the database:

  • MessagePack data format and MessagePack based client-server protocol.
  • Two data engines: 100% in-memory with complete WAL-based persistence and an own implementation of LSM-tree, to use with large data sets.
  • Multiple index types: HASH, TREE, RTREE, BITSET.
  • Document oriented JSON path indexes.
  • Asynchronous master-master replication.
  • Synchronous quorum-based replication.
  • RAFT-based automatic leader election for the single-leader configuration.
  • Authentication and access control.
  • ANSI SQL, including views, joins, referential and check constraints.
  • Connectors for many programming languages.
  • The database is a C extension of the application server and can be turned off.

Supported platforms are Linux (x86_64, aarch64), Mac OS X (x86_64, M1), FreeBSD (x86_64).

Tarantool is ideal for data-enriched components of scalable Web architecture: queue servers, caches, stateful Web applications.

To download and install Tarantool as a binary package for your OS or using Docker, please see the download instructions.

To build Tarantool from source, see detailed instructions in the Tarantool documentation.

To find modules, connectors and tools for Tarantool, check out our Awesome Tarantool list.

Please report bugs to our issue tracker. We also warmly welcome your feedback on the discussions page and questions on Stack Overflow.

We accept contributions via pull requests. Check out our contributing guide.

Thank you for your interest in Tarantool!

tarantool's People

Contributors

alyapunov avatar amdrozdov avatar avtikhon avatar bigbes avatar curiousgeorgiy avatar cyrillos avatar drewdzzz avatar evgenymekhanik avatar georgykirichenko avatar gerold103 avatar gumix avatar igormunkin avatar imeevma avatar korablev77 avatar kostja avatar kshcherbatov avatar kyukhin avatar ligurio avatar locker avatar mejedi avatar nshy avatar pgulutzan avatar pmwkaa avatar rtokarev avatar rtsisyk avatar sergepetrenko avatar totktonada avatar tsafin avatar unera avatar ylobankov avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

tarantool's Issues

box.select selects only one tuple

box.select lua procedure takes only one key so it returns only one tuple while protocol allows to pass N keys to fetch N tuples.

Maybe it is reasonable to extend box.select function, or add box.select_many to add multiselect feature? Or pull request is preferable? ;)

wrong destructor in box.socket

-- init.lua

function server()

    ms = box.socket.tcp()
    ms:bind('127.0.0.1', 8080)
    ms:listen()

    while true do
        local s = ms:accept( .5 )
        if s ~= 'timeout' then
            print("accepted connection ", s)

            s:send('Hello world')
            s:shutdown(box.socket.SHUT_RDWR)
--             s:close()
        end
    end

end

box.fiber.wrap(server)

Point that s:close is commented (for example user forgot do that)

The server crashes at the second connect to localhost:


apache:[~]$ telnet localhost 8080
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Hello world^]
telnet> 
^]
telnet> Connection closed.
apache:[~]$ telnet localhost 8080
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.


log:


GNU gdb (GDB) 7.6 (Debian 7.6-5)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/unera/work/tarantool-triggers/src/box/tarantool_box...done.
[New LWP 19004]
[New LWP 19007]
[New LWP 19008]

warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `tarantool_box: primary pri: 15548 sec: 39425 adm: 10123                       '.
Program terminated with signal 6, Aborted.
#0  0x00007f922a7001e5 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56      ../nptl/sysdeps/unix/sysv/linux/raise.c: Нет такого файла или каталога.
(gdb) #0  0x00007f922a7001e5 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f922a703398 in __GI_abort () at abort.c:90
#2  0x0000000000446b3b in assert_fail (
    assertion=0x50a5a8 "fiber != NULL && fiber->f != NULL && fiber->fid != 0", 
    file=0x50a4d8 "/home/unera/work/tarantool-triggers/src/fiber.cc", 
    line=380, function=0x50a6eb <fiber_loop(void*)::__FUNCTION__> "fiber_loop")
    at /home/unera/work/tarantool-triggers/src/util.cc:328
#3  0x000000000043edbc in fiber_loop (data=0x0)
    at /home/unera/work/tarantool-triggers/src/fiber.cc:380
#4  0x000000000046b9c8 in coro_init ()
    at /home/unera/work/tarantool-triggers/third_party/coro/coro.c:95
(gdb) quit


Disk I/O statistics

Similar to box.stat.info(), provide disk I/O statistics.

Disk I/O statistics should provide statistics bytes read/written for every currently open file.

replication/consistent.test.py fails sporadically

replication/consistent.test.lua [ fail ]
Test failed! Result content mismatch:
--- replication/consistent.result Tue Oct 1 18:36:41 2013
+++ /home/kostja/work/tarantool-master/test/replication/consistent.reject Fri Oct 4 17:49:21 2013
@@ -66,16 +66,8 @@
--# set connection replica
_select(1, 10)


-- [1, 'master - 1']
-- [2, 'master - 2']
-- [3, 'master - 3']
-- [4, 'master - 4']
-- [5, 'master - 5']
-- [6, 'master - 6']
-- [7, 'master - 7']
-- [8, 'master - 8']
-- [9, 'master - 9']
-- [10, 'master - 10']
+- error: '[string " function _select(_begin, _end) a = {} ..."]:1: wrong number

  • of arguments to ''insert'''
    ...
    --# set connection default

Replica panics if fails to download snapshot from master

kostja@olah ~/work/tarantool-master/test/var
% cat tarantool.cfg
slab_alloc_arena = 0.1

pid_file = "tarantool.pid"
bind_ipaddr="INADDR_ANY"
custom_proc_title="replica"

primary_port = 33113
secondary_port = 33114
admin_port = 33115

replication_port=33117
replication_source = 127.0.0.1:33017
kostja@olah ~/work/tarantool-master/test/var
% ./tarantool_box
2013-10-13 12:59:29.437 [25396] 1/sched C> version 1.6.0-306-g1663667
2013-10-13 12:59:29.437 [25397] 1/spawner@replica C> initialized
2013-10-13 12:59:29.440 [25396] 1/sched I> recovery start
2013-10-13 12:59:29.440 [25396] 1/sched I> found an empty data directory, initializing...
2013-10-13 12:59:29.440 [25396] 1/sched I> downloading snapshot from master 127.0.0.1:33017...
2013-10-13 12:59:29.440 [25396] 1/sched exception.cc:74 E> SystemError in connect, called on fd 6, aka 0.0.0.0:47123: Connection refused
2013-10-13 12:59:29.440 [25396] 1/sched F> fatal error, exiting the event loop
2013-10-13 12:59:29.441 [25397] 1/spawner@replica I> Exiting: master shutdown
kostja@olah ~/work/tarantool-master/test/var
% ./tarantool_box --version
Tarantool 1.6.0-306-g1663667

Use delayed garbage collection in slab arena to reduce snapshot memory footprint and impact on RPS & latency

On snapshot, increase snapshot version id (u32),
save current snapshot version within a tuple.

On tuple free, check if tuple version is equal to current
snapshot id, if so do direct free, otherwise append tuple
to the delayed free list. Use the first 8 bytes of tuple as a pointer
in the single-linked delayed free list.

Dispatch with the delayed free list at allocation time, when
snapshot is taken.

Change the mapping of slab arena from private to shared,
so the snapshot process sees the same arena as the master
process.

Given all of the above, the amount of pages which are subject
to COW when forking for box.snapshot() is significantly reduced.
Iteration over tuples in the snapshot process is still functional
and consistent, since only data in the tuple after &tuple->bsize
is used for snapshot. Since index memory is still COW, the
snapshot process preserves a correct view of data.

onexit() is not portable

gh-36 defines snapshot_exit() which uses on_exit(). on_exit() is not standard and is not available on FreeBSD. atexit() can not be used instead, since it doesn't have access to exit() return value.
Change the entire code flow so that we can use _exit() at the end, instead of redefining the chain of atexit() handlers.

tarantool --cat format is not grep friendly

When we had server-side cat, the format was as follows:

lsn:5 tm:1297887999.555 t:65534 127.0.0.1:38720 REPLACE n:04: {'fourth tuple'}

Notice:

  • space no is present
  • the output is one row per line
  • client cookie is not NULL and client address is printed out.

In 1.4.9 the server-side cat was removed, while the client-side cat format is
different and is not grep friendly:

% tarantool --cat test/box/unfinished.xlog
Insert lsn: 5, time: 1297887999.555123, len: 42
['4', 'fourth tuple']

Please restore the old format in tarantool --cat (or make it better than before, but not worse).

FreeBSD

HI

According to the README its been tested on x86/FreeBSD - so was wondering how I can go about compiling it? As I get the following error:

"Error expanding embedded variable." when trying to run "make _release_box/tarantool_silverbox"

thx

NUM64 do not work (tarantool-php)

PHP Fatal error: Uncaught exception 'Exception' with message 'insert failed: 9730(0x00002602): Supplied key field type does not match index type: expected NUM64'

PHP Test Call:
$tuple = array(123, 5);
$res = $tnt->insert(0, $tuple);

space[0].index[0].key_field[0].type = NUM
Work

space[0].index[0].key_field[0].type = NUM64
Do not work, see above.

tarantool console incorrectly prints some strings

Reproduceable with 1.4.9.

When inserting tuples with string data that contains numbers from [1000,9999] range afterwards the same data is printed incorrectly in the shell if I select corresponding tuples. As far as I can tell the actual data in the storage is correct (because I can see correct values if I use python client library to retrieve it)

I.e. example from console session; the following selects end up in incorrect data printed

localhost> insert into t0 values ('1', '1000')
Insert OK, 1 rows affected
localhost> select * from t0 where k0='1'
Select OK, 1 rows affected
[1, 808464433]
localhost> replace t0 values ('1', '2000')
Insert OK, 1 rows affected
localhost> select * from t0 where k0='1'
Select OK, 1 rows affected
[1, 808464434]
localhost> replace t0 values ('1', '9999')
Insert OK, 1 rows affected
localhost> select * from t0 where k0='1'
Select OK, 1 rows affected
[1, 960051513]

but these selects look ok

localhost> replace t0 values ('1', '10000')
Insert OK, 1 rows affected
localhost> select * from t0 where k0='1'
Select OK, 1 rows affected
[1, 10000]
localhost> replace t0 values ('1', '100')
Insert OK, 1 rows affected
localhost> select * from t0 where k0='1'
Select OK, 1 rows affected
[1, 100]

Space triggers

Add a Lua binding to be able to define a trigger run on a change in a space. The trigger is fired after a change is made in a space and receives both old and new tuple.

Race condition running init.lua

init.lua is executed before the server binds to primary/secondary ports, but can insert data, create connections, etc. - in other words, requires a running event loop.

In fact there is no running event loop, and first and any yield in init.lua script takes it off execution and the script is only resumed after general loading is complete, as part of the main server loop.

This creates ugly race conditions, when parts of init.lua are executed concurrently with insert/update/delete requests on the server port.

How to repeat:

add box.fiber.sleep(10000) into init.lua and observe the server doesn't sleep,
but starts and runs

Or:
add box.fiber.sleep() followed by box.insert(), and observe that at start
there is no tuple inserted into the space, i.e. init.lua is not finished before the main loop begins.

How to fix:

In tarantool_lua_load_init_script run the event loop until the init script completes. Then break the loop, finish bootstrap and resume the loop.

FreeBSD: core dump static build

Doesn't work static build on FreeBSD:

[test] 11:02 %> gdb tarantool_box
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
(gdb) r -c tarantool.cfg.sample
Starting program: /usr/local/bin/tarantool_box -c tarantool.cfg.sample
[New LWP 101079]
[New LWP 101079]
/usr/local/bin/tarantool_box: updating a stale pid file
2013-05-18 11:02:39.252 [14549] 1/sched I> space 0 successfully configured
2013-05-18 11:02:39.252 [14549] 1/sched I> recovery start
2013-05-18 11:02:39.252 [14549] 1/sched I> recover from `./00000000000000000001.snap'
2013-05-18 11:02:39.252 [14549] 1/sched I> snapshot recovered, confirmed lsn: 1
2013-05-18 11:02:39.252 [14549] 1/sched I> done `./00000000000000000002.xlog' confirmed_lsn: 5
2013-05-18 11:02:39.252 [14549] 1/sched I> WALs recovered, confirmed lsn: 5
2013-05-18 11:02:39.252 [14549] 1/sched I> building secondary indexes
2013-05-18 11:02:39.252 [14549] 1/sched I> bound to port 33013
2013-05-18 11:02:39.253 [14549] 1/sched I> I am primary
2013-05-18 11:02:39.253 [14549] 1/sched I> bound to port 33014
2013-05-18 11:02:39.253 [14549] 1/sched I> bound to port 33015
[New Thread 800c29400 (LWP 101079/tarantool_box)]

Program received signal SIGABRT, Aborted.
[Switching to Thread 800c29400 (LWP 101079/tarantool_box)]
0x00000000005dedcc in thr_kill ()
(gdb) thread apply all bt
[New Thread 800c29c00 (LWP 113108/tarantool_box)]

Thread 3 (Thread 800c29c00 (LWP 113108/tarantool_box)):
#0  0x000000000052634c in _umtx_op_err ()
#1  0x00000000005206e5 in _thr_umtx_timedwait_uint ()
#2  0x0000000000528edd in cond_wait_common ()
#3  0x000000000044fab7 in wal_writer_pop (writer=0xa06320, input=0x7fffffbfdf90) at fio.h:170
#4  0x00000000004500e2 in wal_writer_thread (worker_args=0x8009e5178) at fio.h:170
#5  0x00000000005281f4 in thread_start ()
#6  0x0000000000000000 in ?? ()
Error accessing memory address 0x7fffffbfe000: Bad address.

Thread 2 (Thread 800c29400 (LWP 101079/tarantool_box)):
#0  0x00000000005dedcc in thr_kill ()
#1  0x000000000062f1b8 in abort ()
#2  0x00000000005d88d5 in uw_init_context_1 (context=0x7fffffffc810, outer_cfa=0x7fffffffca40, outer_ra=0x46ea95) at unwind-pe.h:155
#3  0x00000000005d8d27 in _Unwind_RaiseException (exc=0x800c07f80) at unwind-pe.h:155
#4  0x000000000046ea95 in err_raise_ext ()
#5  0x000000000046eaf2 in lj_err_throw ()
#6  0x000000000046f426 in lj_err_lex ()
#7  0x0000000000485aee in lj_lex_error ()
#8  0x0000000000485c39 in err_token ()
#9  0x000000000048d5fe in lj_parse ()
#10 0x0000000000482a35 in cpparser ()
#11 0x00000000004ab06e in lj_vm_cpcall ()
#12 0x0000000000482bb9 in lua_load ()
#13 0x00000000004a3547 in luaL_loadbuffer ()
#14 0x00000000004a359a in luaL_loadstring ()
#15 0x000000000042dda8 in tarantool_lua_dostring (L=0x109e5378, str=0x668118 "os.execute = nil\nos.exit = nil\nos.rename = nil\nos.tmpname = nil\nos.remove = nil\nrequire = nil\n")
    at _ctype.h:125
#16 0x000000000042e495 in tarantool_lua_sandbox (L=0x109e5378) at _ctype.h:125
#17 0x000000000042e52d in tarantool_lua_load_init_script (L=0x109e5378) at _ctype.h:125
#18 0x000000000042ba39 in main (argc=1, argv=0x7fffffffd610) at rlist.h:142

tarantool.cfg.sample:

slab_alloc_arena = 0.1
pid_file = "box.pid"
primary_port = 33013
secondary_port = 33014
admin_port = 33015
rows_per_wal = 50000
space[0].enabled = 1
space[0].index[0].type = "HASH"
space[0].index[0].unique = 1
space[0].index[0].key_field[0].fieldno = 0
space[0].index[0].key_field[0].type = "NUM"
work_dir = "var"

Improve speed of yaml emitter

It takes over 1 second on my box to output a large Lua table (100k records) to the administrative console. The bottleneck is most likely in the new yaml emitter. In particular, the emitter allocates memory for every node with malloc, so it performs 100k malloc calls.

How it could possibly be improved:

  • replace malloc with palloc
  • look at other bottlenecks and see what can be improved in yaml token analysis

tarantool crashes if call box.raise through network

test case:

#!/usr/bin/perl

use warnings;
use strict;
use utf8;
use open qw(:std :utf8);
use lib qw(lib ../lib);

use Test::More tests    => 15;
use Encode qw(decode encode);


BEGIN {
    use_ok 'File::Spec::Functions', 'catfile', 'rel2abs';
    use_ok 'File::Basename', 'dirname', 'basename';
    use_ok 'DR::Tarantool::StartTest';
    use_ok 'Coro';
    use_ok 'Coro::AnyEvent';
    use_ok 'DR::Tarantool', 'coro_tarantool';
}

use feature 'state';

my $cfg = rel2abs catfile dirname(dirname __FILE__), 'tarantool.cfg';
ok -r $cfg, "-r $cfg";

my $sh1 = run DR::Tarantool::StartTest(cfg => $cfg);

diag $sh1->log unless ok $sh1->started, 'Shard1 is started';


my $t1 = eval { coro_tarantool port => $sh1->primary_port };
diag $sh1->log unless
    ok $t1, 'Connector to shard1';
ok $t1->ping, 'ping the first shard';

eval {
    $t1->call_lua('box.dostring', [qq{
        box.raise(12345, "test error message")
    }]);
};

like $@, qr{test error message}, 'error message';

crashlog:


Program terminated with signal 6, Aborted.
#0  0x00007fda4dc4d1e5 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56  ../nptl/sysdeps/unix/sysv/linux/raise.c: Нет такого файла или каталога.
(gdb) #0  0x00007fda4dc4d1e5 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007fda4dc50398 in __GI_abort () at abort.c:90
#2  0x00000000004375fc in sig_fatal_cb (signo=<optimized out>)
    at /home/unera/work/deb/tarantool/tarantool-1.5.1.76.g5ed3fbe/src/tarantool.cc:429
#3  <signal handler called>
#4  tnt_errcode_val (errcode=<optimized out>)
    at /home/unera/work/deb/tarantool/tarantool-1.5.1.76.g5ed3fbe/include/errcode.h:136
#5  iproto_reply_error (out=out@entry=0x7fda030d8198, 
    req=req@entry=0x7fda4eced038, e=...)
    at /home/unera/work/deb/tarantool/tarantool-1.5.1.76.g5ed3fbe/src/iproto.cc:630
#6  0x000000000045599d in iproto_reply (header=0x7fda4eced038, 
    out=0x7fda030d8198, callback=<optimized out>, port=0x7fda4ee16ee0)
    at /home/unera/work/deb/tarantool/tarantool-1.5.1.76.g5ed3fbe/src/iproto.cc:654
#7  iproto_process_request (request=<optimized out>)
    at /home/unera/work/deb/tarantool/tarantool-1.5.1.76.g5ed3fbe/src/iproto.cc:675
#8  0x0000000000455243 in iproto_queue_handler(typedef __va_list_tag __va_list_tag *) (ap=<optimized out>)
    at /home/unera/work/deb/tarantool/tarantool-1.5.1.76.g5ed3fbe/src/iproto.cc:231
#9  0x0000000000445aa0 in fiber_loop (data=<optimized out>)
    at /home/unera/work/deb/tarantool/tarantool-1.5.1.76.g5ed3fbe/src/fiber.cc:383
#10 0x0000000000465e57 in coro_init ()
    at /home/unera/work/deb/tarantool/tarantool-1.5.1.76.g5ed3fbe/third_party/coro/coro.c:95
(gdb) quit
# Looks like you planned 15 tests but ran 11.
# Looks like you failed 1 test of 11 run.


Support for capped collections

Hi all,

In MongoDB there is a feature called capped collections which are fixed-size lists that support high-throughput operations that insert, retrieve, and delete values based on insertion order.

It is really nice feature that can be used in

  • logging window that shows last X logs
  • most X recent active users in a time windows (do not to store them all)
  • last X activities
  • Newest articles in the front page

Tarantool/Box supports lists and set already which are quite useful to capped collections

Is there any efficient way to do it in Tarantool/Box?

Example code in C connector tp.h is broken

See the test program at https://gist.github.com/IlyaM/5715139 which is based on documentation in comments in tp.h. The test program does a query in space 0 by string key "0e72ae1a-d0be-4e49-aeb9-aebea074363c" . If the test database does contain some tuple with such key then the program ends up hanging waiting for a reply from tarantool. The underlying bug is caused by wrong call to tp_ensure inside of while loop which is written exactly as suggested in the documentation. In such program tp.h ends up allocating insufficient memory for the next read from network what causes the program:
a) to potentially corrupt memory because read() call reads more bytes then was allocated in the buffer;
b) to set internal pointers in tp struct such that the code in while loop thinks it needs to read more data from network when in fact the whole reply have been read.

If I change tp_ensure line to the following then the code does seem to work correctly:

ssize_t new_size = tp_ensure(&rep, to_read + tp_size(&rep));

What suggests that the example in the documentation is wrong.

P.S. Also I'd change example in comments for tp_select to pass non-zero value for limit parameter. I've spent significant time trying to understand why I'm not getting any results because of this.

Make test (and other python code) Python3 compatible

Python3 is the next major version and some Linux distros (e.g. Linux Arch) are already use it as a default. It makes sense to make the code python3 compatible. In theory conversion should be simple especially if you use tools like py2to3.

box.httpd

It would be nice to have box.httpd server to receive http requests inside tarantool.

session storage

It would be nice to have a lua table that was emptyed by disconnect event.

box.session.storage.key = value
box.session.storage.foo = function() ... end

A driver could use the storage for its internal features.

make snapshotting more memory efficient

Allocate slab arena in shared memory to reduce pages splits during snapshotting.

The task is to adaptate patch made by Yuri Nevinitsin @nevinitsin on the branch odkl---shared-arena---index-info---slab-stats---fixes

Confusing "Could not find 'script_dir' option" error message

Hello,

I've just downloaded ubuntu package for tarantool as I wanted to test this database. When trying to start tarantool with sample config (I've copied /etc/tarantool/instances.available/example.cfg to /etc/tarantool/instances.enabled/) it fails with confusing error message:

ilya@denmark:~$ sudo /etc/init.d/tarantool start
tarantool: Staring instances
/usr/bin/tarantool_box: check_config FAILED

  • Could not find 'script_dir' option

Interestingly enough the message goes away if I comment out script_dir line from the config so the error text is obviously incorrect: I would expect such error text if script_dir was required and was missing in the config. Here the error is happening because somehow tarantool is not liking the value of script_dir in my config but the error mesage doesn't explain why (i.e. for example creating an empty directory on script_dir path doesn't remove the error).

OSX always detected as i386

I tried build on:

Darwin dmt.local 12.5.0 Darwin Kernel Version 12.5.0: Mon Jul 29 16:33:49 PDT 2013; root:xnu-2050.48.11~1/RELEASE_X86_64 x86_64

but, I always got results with detect my Darwin box as i386 although it's amd64:

-- VERSION: 1.5.1-119-gbb79c13
-- BUILD: Darwin-i386-Rel
-- C_COMPILER: /usr/bin/clang
-- CXX_COMPILER: /usr/bin/clang++

Online upgrade

Move slab arena to POSIX shared memory.
Make it possible to attach to the memory with a new process, then detach the old one, and
thus provide an online upgrade procedure.

print(64-bit number) adds extra newline

print() of 64-bit number adds unnecessary newline, regardless of where the number is used.

Example:

localhost> lua print(1, 1)

11
...

localhost> lua print(tonumber64(1), 1)

1
1
...

Temporary spaces

A temporary space is one which is empty on every server restart. Data written to such space is not written to the write ahead log, to the snapshot, and is not replicated.

To create a temporary space, specify 'temporary' among space options.

cmake . -DENABLE_GPROF=true && make && make test -> error in swap.test

File "/opt/local/work/tarantool/test/lib/server.py", line 69, in cleanup
shutil.rmtree(self.vardir)
File "/usr/lib/python2.7/shutil.py", line 254, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/usr/lib/python2.7/shutil.py", line 252, in rmtree
os.rmdir(path)
OSError: [Errno 39] Directory not empty: '/home/kostja/work/tarantool/test/var/replica'

box.raise() via iproto can crash the server

localhost> call box.dostring('box.raise(10213, "dsfas")')
1:1 call failed
reconnected
reconnect: Connection refused (errno: 111)
Segmentation fault
Current time: 1380204407
Please file a bug at http://bugs.launchpad.net/tarantool
Attempting backtrace... Note: since the server has already crashed, 
this may fail as well
#0  0x439df0 in sig_fatal_cb(int)+112
#1  0x7fdd10c76af0 in ?
#2  0x45816d in iproto_process_request(iproto_request*)+477
#3  0x457a03 in iproto_queue_handler(__va_list_tag*)+99
#4  0x448180 in fiber_loop(void*)+64
#5  0x4686b7 in coro_init+55
#6  0x413bf0 in ?
Aborted

Confirmed on 1.5.x and 1.6.x.

crash in c connector (tp.h) when building requests with no inital buffer

See the example code below. If I don't use static buffer and instead rely on the library code to allocate it for me then the code crashes without extra call for tp_ensure():

#include "tp.h"

int main() {
    struct tp req;
    tp_init(&req, NULL, 0, tp_realloc, NULL);
    tp_ensure(&req, 100); /* if this line is commented out the the code crashes in call to tp_tuple */
    tp_select(&req, 0, 0, 0, 0);
    tp_tuple(&req);
    tp_sz(&req, "key");

    free(req.s);

    return 0;
}

Server side delimiter in admin console

---
- Tarantool 1.6.0-165-gf63240d
- 'Uptime: 13'
...
localhost> setopt delimiter=';'
localhost> function add_element()
        ->     lol[123] = 'bad table'
        -> end;

---
...
localhost> add_element();

---
- error: '[string "function add_element()     lol[123] = ''bad ta..."]:1: attempt
    to index global ''lol'' (a nil value)'
...

When we create and use bad lua function from client - we get error message, that tells us that error is on the first line, but it's on second.
We need to show right line number.

Tarantool does not response to Memcached-protocol-based request

Hi,

I enabled Tarantool Memcached binrary protocol support.

ps -eaf | grep taran
root      8776  7561  0 08:53 pts/1    00:00:00 tarantool_box: primary pri: 33013 sec: 33014 memc: 33016 adm: 33015

As you can see, 33016 will serve Memcached binary protocol

Here is the client:

import pylibmc


client = pylibmc.Client(["127.0.0.1:33016"], binary=True)
print 'Set: ', client.set('a', '1')
print 'Get: ', client.get('a')

However, no response can be received from Tarantool 1.5.1 after running the script. The script is unable to exit or complete.

snap_io_rate_limit is affected by filesystem cache

snap_io_rate_limit only partially works since data is not flushed on disk. So the OS may choose to accumulate lots of writes and then write them all in bulk, regardless of IO rate limit.

If snap_io_rate_limit is provided, tarantool needs to flush data manually once in a while, to prevent this condition.

Add an option to disable Lua sandbox

I think we should add an option to enable/disable Lua sandbox (with default value set to disabled). The user can decide when sandbox is needed and when is not.

Pager in tarantool command line client (\P)

One of my favorite uses of the MySQL pager:
mysql> \P egrep "inserts|rows"
PAGER set to 'egrep "inserts|rows"'
mysql> show engine innodb status\G
Number of rows inserted 9703525, updated 18, deleted 16, read 57370
46758.47 inserts/s, 0.00 updates/s, 0.00 deletes/s, 21.46 reads/s
1 row in set (0.00 sec)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.