flightaware / speedtables Goto Github PK

We accidentally wrote two shared memory speedtable definitions that referenced two different memory-mapped files. Speedtables accepted this. The result was that the second table behaved like a clone of the first one.

Please make speedtables enforce that all tables shared by the same program reference the same mapped file, else make it work properly with multiple mapped files, which is probably a nonstarter.

Thanks.

If you start up a few separate processes concurrently that share the same table definition (and working area) sometimes the locking during table init fails outright like this:

error deleting "/var/tmp/adsb/c_aircraft_registry_faa.lock.lock": no such file or directory
    while executing
"file delete $name.lock"
    (procedure "unlockfile" line 20)
    invoked from within
"unlockfile $lockfile"
    (procedure "lockfile" line 71)
    invoked from within
"lockfile $build_dir err"
    (procedure "::stapi::init_ctable" line 30)
    invoked from within
"::stapi::init_ctable aircraft_registry_faa "" "modescode IS NOT NULL AND active IS TRUE" modescode nnumber"
    (procedure "create_stapi_tables" line 12)
    invoked from within
"create_stapi_tables"
    (procedure "main" line 88)
    invoked from within

Additionally, sometimes in the same sort of situation, compilation will fail with bizarre errors because there are two processes trying to compile in the same place (i.e. the locking is not providing mutual exclusion).

The _dirty field does not seem to get automatically updated when a record is modified. Is it supposed to be manually set ? Documentation implies it only needs to be manually cleared. Also I couldn't see anywhere in the generated code where the field is set implicitly.

Sample:

(ctable) 56 % puts withDirty:$ctable::withDirty
speedtables Testdirty 0.1 {
table Dirty {
varstring s
}
}

package require Testdirty
withDirty:1

(ctable) 60 % Dirty create d
d
(ctable) 61 % d set 0 {s "foo"}
(ctable) 62 % d search -compare {{= _dirty 1}}
0
(ctable) 64 % d get 0 s _dirty
foo 0
(ctable) 65 % d set 0 {s "bar"}
(ctable) 66 % d get 0 s _dirty
bar 0

The TSV cachefile gets written once during ::stapi::open_cached, but then it never appears to get updated, even if you're calling ::stapi::refresh_ctable.

So for long-running processes that periodically refresh, the cache rarely gets used.

When shared-memory becomes full and a new record is added, then the code currently calls the xxx_shmpanic() handler, which invokes panic() and aborts the process.

The code should be changed to raise a Tcl error that the user can "catch" and handle appropriately, perhaps by deleting some rows from the table.

The following command will crash speedtables because match is being used against an integer (long) field:

inflight search -compare "{match arrivalTime 123456}"

The following command works, without crashing:

inflight search -compare "{= arrivalTime 123456}"

Either "match" should be made to work against integer fields, or it should return an error if you attempt to use that in a query.

(I hesitate to call this a bug, but it causes very non-obvious failures)

If you call stapi::init twice with different -dir values, then any ctables managed by stapi that were created between the two calls get confused because they can no longer find their working files (as [workname $ctable foo] starts returning a different value)

It might make sense to store the build dir used during init_ctable per ctable, so that later changes to build_root do not make things fail.

search fails to cleanly unset the data array on each iteration of the code block. It only reliably clears out array elements that exist in the underlying speed table, but neglects to clear any array elements that were added during the execution of the code block.

Assuming a table containing a single field "id" as a numeric index, this can be reproduced like this:

$speedtable search -array buf -code {
    if {$buf(id) == 5} { 
        set buf(interloper) "this field should only be in record number 5"
    }
    puts [array get buf]
}

Which runs with this output:
id 1
id 2
id 3
id 4
id 5 interloper {this field should only be in record number 5}
id 6 interloper {this field should only be in record number 5}
id 7 interloper {this field should only be in record number 5}
id 8 interloper {this field should only be in record number 5}
etc . . .

I expected the buf array to be reset completely each time the code block is evaluated.

Using search -get with an explicitly specified -fields does not always return the requested fields.

#!/usr/local/bin/tclsh8.5

package require speedtable

CExtension cowseye 1.0 {
    CTable Infield {
        varstring alpha
        varstring beta
        varstring delta
        varstring gamma
    }
}

package require Cowseye


Infield create infield

infield store [list alpha alfa beta bravo delta delta gamma golf]

infield search -fields {beta gamma} -get foo -code {
    puts $foo
}

Running this should display "bravo golf", but i'm getting "alfa bravo" from it right now.

The != operator appears to behave incorrectly when there are indexed columns present. In the following example, only one record should be returned.

Commenting out the line "infield index create ident" restores correct behavior.

#!/usr/local/bin/tclsh8.5

package require ctable

CExtension cowseye 1.0 {


CTable Infield {
    key fp
    varstring ident indexed 1

    varstring orig notnull 1 default "moo"  indexed 1
    varstring dest notnull 1 default "moo" indexed 1

    long clock notnull 1
}

}


package require Cowseye


Infield create infield
infield index create ident
infield index create orig
infield index create dest


set myflight [list ident N1615A  fp N1615A-airline-12341252  clock 1253246234]
infield store -nocomplain $myflight

set myflight [list ident N1615A  fp N1615A-airline-12342223  clock 1253446234]
infield store -nocomplain $myflight

set myflight [list ident N857CP  fp N857CP-airline-35412351  clock 12342353421]
infield store -nocomplain $myflight



infield search -compare [list [list = ident N1615A] [list != fp N1615A-airline-12342223] [list > clock 1234]] -array birdflight -code {
    parray birdflight
    puts "----"

    if {$birdflight(ident) != "N1615A" || $birdflight(fp) == "N1615A-airline-12342223"} {
        puts "BAD!"
    }
}

During the Boost reimplementation in the cpp branch, support for the "flags" argument to mmap() had to be removed because Boost did not support overriding it at the time.

However, Boost Version 1.54.0 now supports platform-specific flags to mapped_region so we should re-add support for it.

https://svn.boost.org/trac/boost/ticket/8030
http://boost.2283326.n4.nabble.com/interprocess-Platform-specific-flags-for-mapped-region-td4642667.html
http://www.boost.org/users/history/version_1_54_0.html

Alongside the "builder_version" and "definition" methods, it would be nice to have a "tables" method which allowed code to programmatically inspect the tablespace defined for a particular speedtable C extension. I envision this working in a manner similar to the existing "fields" method which is available for each table in the extension.

Something that just returns a Tcl list of table names would be really useful.

The cpp branch will happily build and install even on a machine where boost-libs are not present. configure or the install process should probably test for this instead of waiting for the user to discover the problem at runtime.

STAPI leaks debugging lines to stdout during normal operation:

2009-11-30 11:42:54 CST 54502 Reading c_tablename0 from stapi/c_tablename.tsv
2009-11-30 11:42:54 CST 54502 Writing c_tablename0 to stapi/c_tablename.tsv

An extra list is introduced around values that are already lists:

positions share set moo "1234 2345 23423"
ok {}
positions share get moo
ok {{1234 2345 23423}}

Non-lists seem to be correctly handled:

positions share set moo 1234
ok {}
positions share get moo
ok 1234

It would be nice if the import_postgres_result method of a speedtable had options to call a callback routine every some number of rows imported. Something like

$table import_postgres_result  $resultHandle -callback update -callbackEvery 1000

Using yajl or whatever, add a direct JSON return capability. It can be a direct C-to-C thing and since type inspection from C is trivial and fast we can generate the JSON with fidelity. The Tcl interpreter wouldn't even need to be invoked for each match.

Something like... $table search -compare [list [list > clock $val]] -json varName

...to get the json into a var. You might want a -jsonfp or something, to write directly to a file handle or socket.

Can the 'tclobj' datatype be used for anything right now? I am specifically looking for command substitution where the value with datatype of 'tclobj' or similar is automatically substituted with the return value of the command that the value was set to (like another ctable search etc, leading to nice nested tables with much more complex data structures). The same functionality is achievable with the '-code' option but that is much more verbose. I am just curious if 'tclobj' can be put to a good use. :)

Thanks

when trying to build a ctable on freebsd 10.1-RELEASE with the latest tcl8.6.3, I get make errors related to conversion from void* to char* when using ckalloc. My guess is that a number of things changed in 8.6 that will need to be accounted for.

From a similar bug in VTK (http://www.paraview.org/Bug/view.php?id=13449):

In tcl8.6:

 # define ckalloc(x) \ 
    ((VOID *) Tcl_Alloc((unsigned)(x))

while tcl8.5 had

 # define ckalloc(x) Tcl_Alloc(x)

Example error output:

In file included from stobj/include/ctable.h:38:0,
                 from stobj/birdseye/birdseye-1.0.cpp:27:
stobj/include/shared.c: In function 'shm_t* map_file(const char*, char*, size_t, int, int)':
stobj/include/shared.c:159:17: error: invalid conversion from 'void*' to 'char*' [-fpermissive]
     p->filename = ckalloc(strlen(file)+1);
                 ^
stobj/include/shared.c: In function 'int doCreateOrAttach(Tcl_Interp*, const char*, const char*, size_t, int, shm_t**)':
stobj/include/shared.c:783:21: error: invalid conversion from 'void*' to 'char*' [-fpermissive]
         share->name = ckalloc(strlen(sharename)+1);
                     ^
In file included from stobj/birdseye/birdseye-1.0.cpp:88:0:
stobj/include/ctable_io.c: In function 'int ctable_quoteString(const char**, int*, int, const char*)':
stobj/include/ctable_io.c:53:17: error: invalid conversion from 'void*' to 'char*' [-fpermissive]
          newptr = ckalloc(maxExpansion * length + 1);
                 ^
In file included from stobj/include/ctable_search.c:22:0,
                 from stobj/birdseye/birdseye-1.0.cpp:89:
stobj/include/speedtableHash.c: In function 'ctable_HashEntry* ctable_InitOrStoreHashEntry(ctable_HashTable*, const char*, ctable_HashEntry*, int, int*)':
stobj/include/speedtableHash.c:181:19: error: invalid conversion from 'void*' to 'char*' [-fpermissive]
         hPtr->key = ckalloc (strlen (key) + 1);
                   ^

If gen_refresh_ctable_sql returns 0 to indicate an error, refresh_ctable does not handle this and tries to execute it as SQL, which fails:

03/17/2016 12:05:15 speedtable refresh of airline_fleets failed: ERROR:  syntax error at or near "0"
LINE 1: 0
        ^

The "share" command gives erroneous results and/or triggers coredumps if its arguments aren't perfect. For instance, "share", generates a weird error. "share attach" generates a coredump.

Also, the share command isn't documented, AFAICT.

If speedtables startup detects that a compiled shared library is out of date but it can't recompile the shared library due to permissions problems, it should load the old shared library instead of erroring out.

Also it might be a good idea to have a switch to allow an app to disable speedtable extension autorecompile entirely.

Hello,
This is not an issue, just a question.
Is is possible for a speedtables data structure itself to also be a speedtables data type, therefore allowing nested data structures?

Thanks

Create a speedtables website on github.
Take down speedtables docs on Sourceforge.
Install speedtables docs in the github speedtables website.

Hi.

I get the following error when trying to compile the example Speed Table definition, i've searched on the net on a way to fix it and all I encounter are pointers to freebsd.. I've updated my glibc but still nothing.. see extra details below

Error:

In file included from stobj/include/ctable.h:37:0,
from stobj/Animinfo/Animinfo-1.1.c:24:
stobj/include/shared.c: In function ‘map_file’:
stobj/include/shared.c:228:17: error: ‘MAP_NOCORE’ undeclared (first use in this function)
stobj/include/shared.c:228:17: note: each undeclared identifier is reported only once for each function it appears in
stobj/include/shared.c: In function ‘parse_flags’:
stobj/include/shared.c:1544:19: error: ‘MAP_NOCORE’ undeclared (first use in this function)
stobj/include/shared.c: In function ‘shareCmd’:
stobj/include/shared.c:1783:32: error: ‘MAP_NOCORE’ undeclared (first use in this function)

Linux details: Linux Mint 11
Linux Arsenie 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

GCC details:

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.5.2/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.5.2-8ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-4.5/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.5 --enable-shared --enable-multiarch --with-multiarch-defaults=x86_64-linux-gnu --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib/x86_64-linux-gnu --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.5 --libdir=/usr/lib/x86_64-linux-gnu --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-gold --enable-ld=default --with-plugin-ld=ld.gold --enable-objc-gc --disable-werror --with-arch-32=i686 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.5.2 (Ubuntu/Linaro 4.5.2-8ubuntu4)

Currently compiled speedtables for both TCL 8.5.11 and 8.6b2.. i get the same error in both versions

Also I couldn"t get it to compile using the --with-pgsql command so I gave up.

Thanks in advance:).

There should be a top-level Makefile or something similar. It's weird to have to cd in turn to each of three directories and do make / make install or whatever.

Boost supports a "flyweight" template that would allow easily implementation of deduplication of string values, which could provide a significant memory size reduction if values tend to be repeated a lot. (Basically allowing your tables to be denormalized but without the full storage overhead.)

http://www.boost.org/doc/libs/1_51_0/libs/flyweight/doc/tutorial/basics.html

The column could be defined as something like:
varstring filename notnull 1 default "" dedupe 1

The docs have this intriguiing comment on data types - "tclobj - a Tcl object, more on this powerful capability later" . I couldn't find any further mention or description other than a couple of lines in NOTES. Is there a more detailed use description anywhere ? Or is source diving required ?

Also, as an aside, speedtables looked appealing enough for me to fork it to port to WIndows (apnadkarni/speedtables). Hope to be able to fold changes back at some point.

Thanks

If a second speedtables C shared library attempts to create a shared memory speedtable, it will malfunction. The previously defined tables will return results from the most recently defined one.

Comments in ctables/shared/shared.c indicate that some static variables need to be shared between C extensions for this to work.

While the ultimate fix would be to take the C files that are #included in each C extension into its own shared library, this will be tricky and I anticipate compatibility problems between autoconf-generated stuff and different operating systems following this path.

A much easier alternative would be to use the associated data (Tcl_GetAssocData) facility of Tcl interpreters to create a small structure containing the previously static variables, have the first extension create it and further extensions get a pointer to it. This should fix the problem.

Not sure if this is a bug or a known limitation. The generated code is not thread safe. I understand that with Tcl's thread/interp model, a speedtable cannot be accessed from multiple threads. However, because of the use of static caches (e.g. for allocation of rows), it is not safe to even access different speedtables from different threads.

Docs don't talk about this much

/Ashok

It seems that using a "return" command inside of the body of a -code block will not stop execution of a proc and immediately return that value back. This example demonstrates the problem for remotely mounted ctables, but it is believed to also affect certain variations of local or shmem ctables also.

package require birdclient

::birdclient::mount_speedtables

proc testret {} {
    set result "empty"
    inflight search -array row -limit 1 -code {
        set result "bad"
        return "good"
    }
    return $result
}
puts "return value is [testret]"

"Normal" speedtables are automatically regenerated if their definition changes or if speedtables itself is updated. Speedtables generated by stapi based on DDL read from PostgreSQL are not automatically regenerated.

This can result in strange errors if, for instance, a new version of Pgtcl is put in place. Stapi-based speedtables will reference the old while classic ctables-based speedtables will reference the new, and strange behaviors result.

Please make stapi C extensions be regenerated if speedtables are updated with a new version.

When more than one shared-memory speedtable is created in a process, data starts getting lost from the inserted rows. Here is a sample program:

#!/usr/local/bin/tclsh8.5

package require Tclx
package require speedtable

CExtension bullseye 1.0 {
    CTable Pasture {
        varstring alpha
        varstring beta indexed 1
        varstring delta
        varstring gamma indexed 1
    }

    CTable Barn {
        varstring alpha
        varstring beta
        varstring delta
        varstring gamma
    }
}

package require Bullseye


if {$argv == 1} {
    puts "using shmem"
    Pasture create mypasture master name "moo3" file "mypasture.dat" size "256M"
    Barn create mybarn master name "moo4" file "mybarn.dat" size "256M"
} else {
    puts "using mem"
    Pasture create mypasture
    Barn create mybarn
}

puts "created"

for {set i 0} {$i < 10000} {incr i} {
    mypasture store [list alpha alfa$i beta bravo$i delta delta$i gamma golf$i]

    if {$i % 150 == 0} {
        mybarn store [list alpha alfa$i beta bravo$i delta delta$i gamma golf$i]
    }
}

puts "inserted"

mybarn search -array barnrow -limit 10 -sort alpha -code {
    set count [array size barnrow]
    if {$count != 5} {
        puts "Error: wrong number of elements (was $count, expected 5)"
    }
    parray barnrow
    puts ""
}

puts info=[mypasture share info]
puts pools=[mypasture share pools]
puts free=[mypasture share free]

Run it with argument of 0 and it will work:

$ ./shmemtwo.tcl 0
using mem
created
inserted
barnrow(_key)  = 0
barnrow(alpha) = alfa0
barnrow(beta)  = bravo0
barnrow(delta) = delta0
barnrow(gamma) = golf0

barnrow(_key)  = 7
barnrow(alpha) = alfa1050
barnrow(beta)  = bravo1050
barnrow(delta) = delta1050
barnrow(gamma) = golf1050

barnrow(_key)  = 8
barnrow(alpha) = alfa1200
barnrow(beta)  = bravo1200
barnrow(delta) = delta1200
barnrow(gamma) = golf1200

barnrow(_key)  = 9
barnrow(alpha) = alfa1350
barnrow(beta)  = bravo1350
barnrow(delta) = delta1350
barnrow(gamma) = golf1350

barnrow(_key)  = 1
barnrow(alpha) = alfa150
barnrow(beta)  = bravo150
barnrow(delta) = delta150
barnrow(gamma) = golf150

barnrow(_key)  = 10
barnrow(alpha) = alfa1500
barnrow(beta)  = bravo1500
barnrow(delta) = delta1500
barnrow(gamma) = golf1500

barnrow(_key)  = 11
barnrow(alpha) = alfa1650
barnrow(beta)  = bravo1650
barnrow(delta) = delta1650
barnrow(gamma) = golf1650

barnrow(_key)  = 12
barnrow(alpha) = alfa1800
barnrow(beta)  = bravo1800
barnrow(delta) = delta1800
barnrow(gamma) = golf1800

barnrow(_key)  = 13
barnrow(alpha) = alfa1950
barnrow(beta)  = bravo1950
barnrow(delta) = delta1950
barnrow(gamma) = golf1950

barnrow(_key)  = 14
barnrow(alpha) = alfa2100
barnrow(beta)  = bravo2100
barnrow(delta) = delta2100
barnrow(gamma) = golf2100

info=
pools=
free=

Run it with 1 and it fails:

$ ./shmemtwo.tcl 1

using shmem
created
inserted
Error: wrong number of elements (was 2, expected 5)
barnrow(_key)  = 0
barnrow(gamma) = alfa0

Error: wrong number of elements (was 2, expected 5)
barnrow(_key)  = 10065
barnrow(gamma) = alfa9998

Error: wrong number of elements (was 2, expected 5)
barnrow(_key)  = 10064
barnrow(gamma) = alfa9997

Error: wrong number of elements (was 1, expected 5)
barnrow(_key) = 10063

Error: wrong number of elements (was 1, expected 5)
barnrow(_key) = 10062

Error: wrong number of elements (was 2, expected 5)
barnrow(_key)  = 10061
barnrow(gamma) = alfa9994

Error: wrong number of elements (was 2, expected 5)
barnrow(_key)  = 10060
barnrow(gamma) = alfa9993

Error: wrong number of elements (was 1, expected 5)
barnrow(_key) = 10059

Error: wrong number of elements (was 1, expected 5)
barnrow(_key) = 10058

Error: wrong number of elements (was 2, expected 5)
barnrow(_key)  = 10057
barnrow(gamma) = alfa9990

info=size 268435456 flags {core nosync shared} name moo4 creator 1 filename mybarn.dat base -1610612736
pools=128 131072 1 131004
free=248225512

I am on 64bit Ubuntu, have the latest TCL 8.6.4 64bit installed, Speedtable compiles but fails "make test" or fails during runtime, with the following error:

% package req speedtable
1.9.0
speedtables Animinfo 1.1 {
table animation_characters {
varstring name indexed 1 unique 0
varstring home
varstring show indexed 1 unique 0
varstring dad
boolean alive default 1
varstring gender default male
int age
int coolness
}
}
In file included from stobj/Animinfo/Animinfo-1.1.cpp:86:0:
stobj/include/ctable_io.c: In function ‘int ctable_quoteString(const char*, int, int, const char_)’:
stobj/include/ctable_io.c:63:36: error: invalid conversion from ‘const char_’ to ‘char_’ [-fpermissive]
char *off = strchr(special, c);
^
In file included from stobj/Animinfo/Animinfo-1.1.cpp:87:0:
stobj/include/ctable_search.c: In function ‘void ctable_performance_callback(Tcl_Interp_, CTable_, Tcl_Obj_ const_, int, timespec_, int)’:
stobj/include/ctable_search.c:2408:20: error: ‘CLOCK_VIRTUAL’ was not declared in this scope
clock_gettime (CLOCK_VIRTUAL, &endTimeSpec);
^
stobj/include/ctable_search.c: In function ‘int ctable_SetupAndPerformSearch(Tcl_Interp_, Tcl_Obj_ const_, int, CTable_, int)’:
stobj/include/ctable_search.c:2512:17: error: ‘CLOCK_VIRTUAL’ was not declared in this scope
clock_gettime (CLOCK_VIRTUAL, &startTimeSpec);

from jima (2011-01-27) on http://wiki.tcl.tk/20026

I have been able to set up Speed Tables for my Linux box. As the authors stated in their documentation their main focus has been BSD OS. I have encountered some tricky points in my way and I wanted to reflect here my steps just in case this could be of value to other people interested in this project.

I hope this will also help authors and contributors to find a better way of packaging this so as to cover BDS and Linux in the same go. Unfortunately I don't know anything about autotools to help in a better way than simply pointing to the differences between both OS'es.

Thanks to all people in the Tcl chat that helped me in my way to success.

The original scheme of things:

• In Speed Tables one can write a Tcl script that is the seed to create a set of (mainly) C files in a build directory. These files are compiled and built into a Tcl extension. This building process is done by a Tcl script which invokes the compiler and linker by means of exec command.

Main difficulties in my Linux (debian lenny) box:

• (I) In general, some options (like using tclsh8.4) seem to be hardcoded in the files that guide the building process.
• (II) The compiler seems to be writing warning to stderr. This causes exec command to stop execution.
• (III) Speed Tables use C function qsort_r that has been developed originally in BSD. This function can be found in the C standard library for Linux but, alas, with a different order used for its parameters (as seen on the web).
• (IV) Tcl would not find the entry point for the extension.

Workarounds:

For (I):

In file ctables/Makefile I had to set PREFIX to my install dir:

• PREFIX=/mnt/tis/ins/src

In file ctables/config.tcl I tweaked the following directives:

• set showCompilerCommands 1
• At the end of the file: set withPgtcl 0

In file ctables/mkConfig.sh I did:

• tcl_versions="8.6"
• tcl_prefixes="/mnt/tis/ins/src/lib"

By the way, instead of letting the Makefile produce the sysconfig.tcl from config.tcl I was somehow forced to do it manually: mkConfig.sh > sysconfig.tcl.

After the generation of sysconfig.tcl I tweaked it to:

• Add to the list of define directives: -DCTABLE_NO_SYS_LIMITS=1
  • Declare the linker in a proper way: set sysconfig(ld) {gcc -shared}

Last but not least:

In file ctables/tests/Makefile I just told the system to use my tclsh instead of a hardwired one:

• TCLSH=tclsh

For (II)

In file ctables/gentable.tcl just tweak a proc called myexec:

• eval exec -ignorestderr -- $command

For (III)

First, I had to update my system to have a libc that included function qsort_r by grabbing latest version of it from debian unstable.

In file ctables/ctable.h:

• I had to load the ugly __USE_GNU directive ahead of stdlib to allow loading of qsort_r.

 #define __USE_GNU
 #include <stdlib.h>

• I had to distinguish between the loading of the BDS version or the Linux one:

 #ifdef JIMA_QSORT_R_BSD
    int (*sort_compare) (void *clientData, const void *pointer1, const void *pointer2);
 #else
    int (*sort_compare) (const void *pointer1, const void *pointer2, void *clientData);
 #endif

In file ctables/ctable_search.c:

This is where the actual call to qsort_r is done so I had to flag here as well:

 #ifdef JIMA_QSORT_R_BSD
      qsort_r (search->tranTable, search->matchCount, sizeof (ctable_HashEntry *), &search->sortControl, creator->sort_compare);
 #else
      qsort_r (search->tranTable, search->matchCount, sizeof (ctable_HashEntry *), creator->sort_compare, &search->sortControl);
 #endif

Not only parameters for qsort_r have different orders for both OS'es. The comparison function being passed to qsort_r have different orders for its parameters too.

This I hacked in a very dirty way in file: ctables/gentable.tcl at the point of variable sortCompareHeaderSource:

// JIMA_NON_BSD
int ${table}_sort_compare(const void *vRow1, const void *vRow2, void *clientData)

For (IV)

In file ctables/init-exten.c-subst:

• Change mentions to EXTERN int by int DLLEXPORT.

TODO's

Obviously this is a very dirty set of workarounds... a better way to do it would require probably find the best point to insert C preprocessor directives and let all code be governed properly by them.

I haven't yet tested the install procedure and checked on shared memory goodies in a detailed way.

While using "search -delete" on a shared-memory table, I'm getting a panic:
PANIC: Trying to free pointer outside mapped memory!

The table was created with:

   InFlight create inflight master name "birdseye.inflight" file "/path/whatever.inflight" panic false size 256M

And the code that is doing the deleting is something like this:

    set query [list [list in fp $delfp]]
    #logger "DEBUG: trimming: $query"
    logger "DEBUG: trimming: count = [inflight count]"
    set inflightTrimmed [inflight search -compare $query -delete 1]

The logging output and debug dump is as follows. The 2nd line ("memory =") is something I added to shared/shared.c to debug the address being accessed. As you can see, the memory is outside of the shared-memory range, and the "data =" line is the text that is at that address. It appears to be the list the arguments that I passed as "$delfp" in the -compare above.

01/10/2011 15:20:01 DEBUG: trimming: count = 2813
memory = 0x800d8c030, shm->map = 0xa0000000, map_size = 268435456
data = CCA1311-1294466541-airline-0173 ASH1015-1294526758-62-0 CCA134-1294466541-airline-0201 ANA884-1294467664-airline-0264 LOF7757-1294449951-81-0 CCA1507-1294466541-airline-0319 BTA2706-1294355100-schedule-0026 SKW667T-1294511756-189-0 CCA1102-1294466541-airline-0032
PANIC: Trying to free pointer outside mapped memory!

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x800d02180 (LWP 100316)]
0x0000000800a33cfc in kill () from /lib/libc.so.7
(gdb) bt
#0  0x0000000800a33cfc in kill () from /lib/libc.so.7
#1  0x0000000800a32b6b in abort () from /lib/libc.so.7
#2  0x0000000801623344 in shmpanic (s=Variable "s" is not available.
) at shared.c:1221
#3  0x0000000801623bdf in shmfree (shm=0x800ed7430, 
    memory=0x800d8c030 "CCA1311-1294466541-airline-0173 ASH1015-1294526758-62-0 CCA134-1294466541-airline-0201 ANA884-1294467664-airline-0264 LOF7757-1294449951-81-0 CCA1507-1294466541-airline-0319 BTA2706-1294355100-schedul"...) at shared.c:769
#4  0x0000000801623cd0 in InFlight_deleteKey (ctable=Variable "ctable" is not available.
) at stobj/birdseye/birdseye-1.0.c:7162
#5  0x00000008016242ce in InFlight_delete (ctable=0xa0000048, vRow=0x801515c00, indexCtl=-1) at stobj/birdseye/birdseye-1.0.c:7192
#6  0x0000000801633749 in ctable_SetupAndPerformSearch (interp=0x800d25800, objv=Variable "objv" is not available.
) at ctable_search.c:2397
#7  0x000000080163e699 in InFlightObjCmd (cData=0xa0000048, interp=0x800d25800, objc=6, objv=0x800d37880) at stobj/birdseye/birdseye-1.0.c:18513
#8  0x0000000800661343 in TclEvalObjvInternal () from /usr/local/lib/libtcl85.so
#9  0x00000008006a7637 in TclExecuteByteCode () from /usr/local/lib/libtcl85.so

I think this argument also should be required else if you have multiple speedtables it causes problems (?).

Currently, attempting to insert too much into a table just causes an unrecoverable abort().

package require speedtable

CExtension bullseye 1.0 {
    CTable Pasture {
        varstring alpha
        varstring beta
        varstring delta
        varstring gamma
    }
}

package require Bullseye


Pasture create mypasture master name "moo" file "mypasture.dat" size 4096

for {set i 0} {$i < 1000} {incr i} {
    mypasture store [list alpha alfa beta bravo delta delta gamma golf]
}

$ ./shmemtest.tcl

Out of shared memory for "mypasture.dat".
Abort (core dumped)

I've got a table I'm caching with ::stapi::init_ctable and ::stapi::open_cached. One of the varchar fields in the backing database table contains a trailing newline.

If no .tsv exists in the ctables directory, open_cached reads everything in the first time without complaint, but if I restart the application and it attempts to use the cached .tsv data it crashes.

Inspecting the .tsv file I see that there's a literal ^M in the file, which breaks the record into two lines.

In FreeBSD 10, the default compiler was switched from GNU gcc to clang. As a result, the g++ command is gone and has been replaced by the c++ command.

This project is using autoconf, but none of the results from that are being used to tune what compiler commands are launched from the Tcl code. For example, ctables/config.log shows:
configure:5027: checking for g++
configure:5057: result: no
configure:5027: checking for c++
configure:5043: found /usr/bin/c++
configure:5054: result: c++

but, ctables/sysconfig.tcl has:
sysconfig.tcl:set sysconfig(cxx) {g++}
sysconfig.tcl:set sysconfig(cxxld) {g++ -shared}

As a result, this does not work on any system that uses something other than gcc/g++.

http://speedtables.sourceforge.net/ch08.html documents that the "size" argument to the "create" command is required, however it does not actually appear to be. If you do not specify it, then this default is assumed:

./ctables/ctable.h:#define DEFAULT_SHARED_SIZE (1024*1024*4)

From ctables/template.c-subst:

                    if(raw_size) {
                        if (!parse_size(raw_size, &share_size)) {
                            Tcl_AppendResult(interp, "Bad size, must be an integer optionally followed by 'k', 'm', or 'g': ", raw_size, NULL);
                            goto createError;
                        }
                    } else {
                        share_size = DEFAULT_SHARED_SIZE;
                    }

Either the documentation is wrong, or the code is wrong.

I am seeing the performance of the "set" and "store" operations take massively increased amounts of time on a shared-memory table after it has been performing operations for several days continuously. I suspect this may be due to shmem heap fragmentation and the difficulty in locating appropriately sized memory blocks.

08/30/2011 01:03:16{43325} position_store consumed 12774 of 6108282 clicks over 72 calls (0.21%, 177/call)
08/30/2011 01:03:22{43325} position_store consumed 1469 of 6024561 clicks over 8 calls (0.02%, 183/call)
08/30/2011 01:03:28{43325} position_store consumed 24610 of 5318539 clicks over 143 calls (0.46%, 172/call)
08/30/2011 01:03:34{43325} position_store consumed 11013 of 6244828 clicks over 63 calls (0.18%, 174/call)
08/30/2011 01:03:40{43325} position_store consumed 8440 of 6058986 clicks over 47 calls (0.14%, 179/call)
08/30/2011 01:03:46{43325} position_store consumed 28877 of 6209391 clicks over 178 calls (0.47%, 162/call)
08/30/2011 01:03:52{43325} position_store consumed 23048 of 6224565 clicks over 132 calls (0.37%, 174/call)
....
08/30/2011 01:28:48{43325} position_store consumed 1120031 of 6016701 clicks over 84 calls (18.62%, 13333/call)
08/30/2011 01:28:54{43325} position_store consumed 1243474 of 6015077 clicks over 96 calls (20.67%, 12952/call)
08/30/2011 01:29:00{43325} position_store consumed 1157939 of 5988499 clicks over 89 calls (19.34%, 13010/call)
08/30/2011 01:29:07{43325} position_store consumed 1130888 of 7485910 clicks over 92 calls (15.11%, 12292/call)
08/30/2011 01:29:13{43325} position_store consumed 1048925 of 5646486 clicks over 77 calls (18.58%, 13622/call)
08/30/2011 01:29:19{43325} position_store consumed 1151757 of 6422342 clicks over 85 calls (17.93%, 13550/call)
08/30/2011 01:29:25{43325} position_store consumed 958299 of 5545420 clicks over 70 calls (17.28%, 13689/call)

After adding some more timing logic within the gentable.tcl, I can see that the increase in runtime is occurring

Position_foc_StoreHashEntry consumed 0 of 128 clicks over 2 calls (0.00%, 0/call)
Position_foc_ListInsertHead consumed 0 of 128 clicks over 2 calls (0.00%, 0/call)
Position_foc_shmalloc1 consumed 0 of 8 clicks over 66 calls (0.00%, 0/call)
Position_foc_shmalloc2 consumed 0 of 8 clicks over 66 calls (0.00%, 0/call)
Position_foc_StoreHashEntry consumed 0 of 8 clicks over 66 calls (0.00%, 0/call)
Position_foc_ListInsertHead consumed 0 of 8 clicks over 66 calls (0.00%, 0/call)
Position_foc_shmalloc1 consumed 0 of 141 clicks over 65 calls (0.00%, 0/call)
Position_foc_shmalloc2 consumed 0 of 141 clicks over 65 calls (0.00%, 0/call)
...
Position_foc_StoreHashEntry consumed 0 of 135 clicks over 10 calls (0.00%, 0/call)
Position_foc_ListInsertHead consumed 0 of 135 clicks over 10 calls (0.00%, 0/call)
Position_foc_shmalloc1 consumed 0 of 115 clicks over 10 calls (0.00%, 0/call)
Position_foc_shmalloc2 consumed 16 of 115 clicks over 10 calls (13.91%, 1/call)
Position_foc_StoreHashEntry consumed 0 of 115 clicks over 10 calls (0.00%, 0/call)
Position_foc_ListInsertHead consumed 0 of 115 clicks over 10 calls (0.00%, 0/call)
Position_foc_shmalloc1 consumed 0 of 135 clicks over 11 calls (0.00%, 0/call)
Position_foc_shmalloc2 consumed 17 of 134 clicks over 11 calls (12.69%, 1/call)
Position_foc_StoreHashEntry consumed 0 of 134 clicks over 11 calls (0.00%, 0/call)

in this function call of gentable.tcl:

    key_value = (char *)shmalloc(ctable->share, strlen(key)+1);

Note that there is another call to shmalloc() just a few lines above that instance in gentable.tcl, but it is for a fixed structure size and my timing logs indicate that its runtime has not measurably increased, so it indeed seems to be related to satisfying the allocation of certain block sizes.

I'll try to instrument shmalloc() with more detailed timing information, but the resolution of the user-land clock is beginning to be restrictive at this narrow of a level.

Karl reports that the base address specified for the memory mapped file may not be actually honored and that a self-assigned address is used instead.

Hello, I have tried using speedtables several times over the years, could never get it to work on Linux.
I am currently on Linux 64bit using the latest 8.6 Tcl source. Speedtables goes through configure/make/make install routine just fine. 'package require speedtable" also works. But after trying to create the Animinfo package using example from SorceForge docs:

package require speedtable

speedtables Animinfo 1.1 {
table animation_characters {
varstring name indexed 1 unique 0
varstring home
varstring show indexed 1 unique 0
varstring dad
boolean alive default 1
varstring gender default male
int age
int coolness
}
}

I get the following error (and the package Animinfo is never created):

In file included from stobj/include/ctable.h:37:0,
from stobj/Animinfo/Animinfo-1.1.c:17:
stobj/include/shared.c: In function ‘parse_flags’:
stobj/include/shared.c:1402:11: warning: variable ‘word’ set but not used [-Wunused-but-set-variable]
stobj/Animinfo/Animinfo-1.1.c: In function ‘animation_characters_sanity_check_pointer’:
stobj/Animinfo/Animinfo-1.1.c:590:3: warning: format ‘%d’ expects a matching ‘int’ argument [-Wformat]

Hopefully this can be resolved.

Thanks

I'm occasionally seeing a segfault in a shared-memory child while it is trying to write out a table to a file.

I have a stacktrace, however this doesn't have full debugging symbols so it is of limited use. I will try to get a more complete stacktrace the next time it occurs.

Program received signal SIGSEGV, Segmentation fault.
0x0000000800a56f27 in strlen () from /lib/libc.so.7
(gdb) bt
 #0  0x0000000800a56f27 in strlen () from /lib/libc.so.7
 #1  0x0000000801825538 in InFlight_get_string () from stobj/birdseye/libbirdseye.so
 #2  0x0000000801828edd in InFlight_dstring_append_get_tabsep () from stobj/birdseye/libbirdseye.so
 #3  0x0000000801824ab6 in ctable_SearchAction () from stobj/birdseye/libbirdseye.so
 #4  0x000000080183fdde in ctable_SetupAndPerformSearch () from stobj/birdseye/libbirdseye.so
 #5  0x000000080184d3b3 in InFlightObjCmd () from stobj/birdseye/libbirdseye.so
 #6  0x0000000800675313 in TclEvalObjvInternal () from /usr/local/lib/libtcl85.so.1
 #7  0x00000008006bb5d7 in TclExecuteByteCode () from /usr/local/lib/libtcl85.so.1
 #8  0x00000008006f8a82 in TclObjInterpProcCore () from /usr/local/lib/libtcl85.so.1
#9  0x0000000800675313 in TclEvalObjvInternal () from /usr/local/lib/libtcl85.so.1
#10 0x00000008006bb5d7 in TclExecuteByteCode () from /usr/local/lib/libtcl85.so.1

In the "set" and "store" methods, if the fieldname contains a leading dash, simply ignore the leading dash. This allows us to use field-value pairs as Itcl and Tk do.

Might want to even make a switch to get field names with a leading dash within search, I don't know.

dropping and recreating index can cause panic

inflight index indexed
ok {ident orig dest clock lat}
inflight index indexable
ok {ident orig dest clock lat}
inflight index drop lat
ok {}
inflight index create lat
Connection closed by foreign host.

The panic produced was:

Double insert row for field lat
Abort

Which occurs in:

ctables/ctable_search.c
// invariant: prev is always NULL if not in list
    if(row->_ll_nodes[index].prev != NULL) {
        panic ("Double insert row for field %s", ctable->creator->fields[field]->name);
    }

If no time ("changed") column is available then refresh_ctable will reread the whole table from the DB.

However it does not appear to clear the current table contents before doing so, so deleted rows will live forever.

Currently shared-memory tables store physical pointer addresses of other records and strings within shared-memory. This has the side effect of necessitating enforcement that attaching to an existing shared-memory table is able to be allocated at the same memory address. In general, it is not possible for the OS to guarantee that the same virtual address location will be available for mmap() in another process so the current method is really only well-suited for use by processes that inherit the table through forking, or independent processes that happen to be lucky when calling mmap.

Switching all absolute pointers to be relative pointers (with respect to the address of the shared-memory block) will have a slight performance penalty but will allow greater flexibility. By using a preprocessor #define around all pointer operations, it would be possible to conditionally switch on/off use of relative pointers at compile-time if we believe the performance hit is significant.

Right now you have to set a variable to tell speedtables if you want PostgreSQL support. This should be an option to the configure script.