syzygy1 / tb Goto Github PK

The statsfiles give examples of frustrated wins for White in '101', i.e., 1 ply. But they do not give examples frustrated wins for Black in '101', i.e. 1 ply.

There are four endgame where this and the omission occurs: KQNPKQ, KQRPKP, KRRPKP and KQPKBP.
For the first three, I easily found exemplar frustrated wins:
KQNPKQ/0-1: 4K1k1/1N1P4/8/3q4/8/1Q6/8/8 b, 1. ... Qxb3 {RdM's KNPKQ/0-1 example, pos 1w}
KQRPKP/0-1: 8/1K6/8/2P5/2R5/7k/p7/1Q6 b, 1. ... axb1=Q+ {KRPKQ: dtz50' = 144p}
KRRPKP/0-1: 8/1K6/8/2P5/2R5/7k/p7/1R6 b (similar), 1. ... axb1=Q+ {KRPKQ: dtz50' = 144p}

I have not found an exemplar longest frustrated win for KQPKBP/0-1 and this now seems to be the only lacuna in the list of exemplar frustrated wins - and so is on my 'much wanted' list.

A first inspection of the stats-p code shows no asymmetry re lcw_ply and glcw_ply v lcb_ply and glcb_ply. Is something asymmetric happening with the integer 'i'?

Sorry I cannot be more help - Guy

Hello, I do have exactly the same error either using MSYS2 or WSL with mingw-w64 libs :

$ make all
make[1]: Entering directory '/home/youri/tb/src'
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -mbmi2 -mwin32 -c threads.c -DBMI2 -DUSE_POPCNT -DTBPIECES=7 -DUSE_ZSTD -DCOMPRESSION_THREADS=6 -DREGULAR -o objsr/threads.o
In file included from threads.c:17:
util.h:7: warning: "min" redefined
    7 | #define min(a,b) ((a) < (b) ? (a) : (b))
      |
In file included from C:/msys64/mingw64/include/windef.h:9,
                 from C:/msys64/mingw64/include/windows.h:69,
                 from threads.c:12:
C:/msys64/mingw64/include/minwindef.h:177: note: this is the location of the previous definition
  177 | #define min(a, b) (((a) < (b)) ? (a) : (b))
      |
threads.c:285:8: error: unknown type name 'pthread_t'
  285 | static pthread_t cmprs_threads[COMPRESSION_THREADS];
      |        ^~~~~~~~~
threads.c:287:8: error: unknown type name 'pthread_barrier_t'
  287 | static pthread_barrier_t cmprs_barrier;

Any idea ? :'(

@syzygy1
Hi Ronald.
Apologies for opening an issue but not sure where best to contact you...

I was wondering if TB generation was CPU speed or memory bandwidth limited? There are now 480GB Intel Optane 900P drives available that could be put into a RAID configuration for a total of almost 1TB and used entirely as a swap file. I wonder if having this 1TB of cheap virtual RAM that may be say 20x slower than normal DRAM would open a genuine possibility of generating 7-men?

Thanks Fisherman

https://github.com/syzygy1/tb/blob/master/src/probe.c#L788

runtime error: implicit conversion from type 'int' of value -1 (32-bit, signed) to type 'unsigned long' changed the value to 18446744073709551615 (64-bit, unsigned)

In readme.md there is no information about tbcheck --compare option.

If you assume one engine is running, memory mapping DTZs creates no problems; however, every serious chess player uses multiple instances of an engine simultaneously. Having one instance memory mapping DTZs that prevents WDL files from getting RAM loaded in a second engine instance is a huge performance hit. DTZs should NEVER be memory mapped. Simple OS caching is more than enough for the DTZ files.

Create a binary file and host this app on GitHub

Hi.

It seems that the memory allocated for struct PairsData in setup_pairs() is never freed after STOP and even after QUIT. Visible e.g. in Arena GUI that shows the engine consuming more and more memory when probing the TB.
Bug or design?
I would suggest some cleanup function that could be called from the engine.

is it possible to make an adapted tb shrinking for an engine :
considerer three parameter :
depth DEFAULT 20,
min threshold DEFAULT 10 centipawn ,
max threshold DEFAULT 100 centipawn,
take the tb and keep only the positions that a 1 thread analysis at fixed depth 20 evaluate less than max threshold and that are winning and the posistions evaluated less than minthreshold and that are DRAW. It should shrink the TB really needed for syzygysf by huge factor if it is possible and still contain all the information an engine might need.
Mehdi

How come it only generates up to d file and not up to h file?
Is it suppose to generate for half the board?

Checksums in the files dtz345.txt, wdl345.txt differ from the checksums calculated by me.
But my checksums do NOT differ from the following checksums:
http://olympuschess.com/egtb/sbases/checksum.md5
http://tablebase.sesse.net/syzygy/6-WDL/3-4-5-6.md5
http://kirill-kryukov.com/chess/tablebases-online/Syzygybases-WDL-3-4-5.md5

[oppie@dell src]$ make
make[1]: Entering directory '/home/oppie/tb/src'
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -c tbgen.c -DMAGIC -DUSE_POPCNT -DTBPIECES=7 -DUSE_ZSTD -DCOMPRESSION_THREADS=6 -DREGULAR -o objsr/tbgen.o
In file included from tbgen.c:93:
rtbgen.c: In function ‘reset_captures_w’:
rtbgen.c:816:11: warning: storing the address of local variable ‘v’ in ‘reset_v’ [-Wdangling-pointer=]
816 | reset_v = v;
| ~~~~~~~~^~~
rtbgen.c:814:11: note: ‘v’ declared here
814 | uint8_t v[256];
| ^
rtbgen.c:741:17: note: ‘reset_v’ declared here
741 | static uint8_t *reset_v;
| ^~~~~~~
rtbgen.c: In function ‘reset_captures_b’:
rtbgen.c:852:11: warning: storing the address of local variable ‘v’ in ‘reset_v’ [-Wdangling-pointer=]
852 | reset_v = v;
| ~~~~~~~~^~~
rtbgen.c:850:11: note: ‘v’ declared here
850 | uint8_t v[256];
| ^
rtbgen.c:741:17: note: ‘reset_v’ declared here
741 | static uint8_t *reset_v;
| ^~~~~~~
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -c permute.c -DMAGIC -DUSE_POPCNT -DTBPIECES=7 -DUSE_ZSTD -DCOMPRESSION_THREADS=6 -DREGULAR -o objsr/permute.o
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -c compress.c -DMAGIC -DUSE_POPCNT -DTBPIECES=7 -DUSE_ZSTD -DCOMPRESSION_THREADS=6 -DREGULAR -o objsr/compress.o
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -c huffman.c -DMAGIC -DUSE_POPCNT -DTBPIECES=7 -DUSE_ZSTD -DCOMPRESSION_THREADS=6 -DREGULAR -o objsr/huffman.o
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -c threads.c -DMAGIC -DUSE_POPCNT -DTBPIECES=7 -DUSE_ZSTD -DCOMPRESSION_THREADS=6 -DREGULAR -o objsr/threads.o
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -c lz4.c -DMAGIC -DUSE_POPCNT -DTBPIECES=7 -DUSE_ZSTD -DCOMPRESSION_THREADS=6 -DREGULAR -o objsr/lz4.o
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -c checksum.c -DMAGIC -DUSE_POPCNT -DTBPIECES=7 -DUSE_ZSTD -DCOMPRESSION_THREADS=6 -DREGULAR -o objsr/checksum.o
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -c city-c.c -DMAGIC -DUSE_POPCNT -DTBPIECES=7 -DUSE_ZSTD -DCOMPRESSION_THREADS=6 -DREGULAR -o objsr/city-c.o
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -c util.c -DMAGIC -DUSE_POPCNT -DTBPIECES=7 -DUSE_ZSTD -DCOMPRESSION_THREADS=6 -DREGULAR -o objsr/util.o
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -o rtbgen objsr/tbgen.o objsr/permute.o objsr/compress.o objsr/huffman.o objsr/threads.o objsr/lz4.o objsr/checksum.o objsr/city-c.o objsr/util.o -lzstd
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: note: see the ‘-flto’ option documentation for more information
make[1]: Leaving directory '/home/oppie/tb/src'
make[1]: Entering directory '/home/oppie/tb/src'
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -c tbgenp.c -DMAGIC -DUSE_POPCNT -DTBPIECES=7 -DUSE_ZSTD -DCOMPRESSION_THREADS=6 -DREGULAR -o objsr/tbgenp.o
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -o rtbgenp objsr/tbgenp.o objsr/permute.o objsr/compress.o objsr/huffman.o objsr/threads.o objsr/lz4.o objsr/checksum.o objsr/city-c.o objsr/util.o -lzstd
lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
lto-wrapper: note: see the ‘-flto’ option documentation for more information
make[1]: Leaving directory '/home/oppie/tb/src'
make[1]: Entering directory '/home/oppie/tb/src'
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -c tbver.c -DMAGIC -DUSE_POPCNT -DTBPIECES=7 -DUSE_ZSTD -DCOMPRESSION_THREADS=6 -DREGULAR -o objsr/tbver.o
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -c decompress.c -DMAGIC -DUSE_POPCNT -DTBPIECES=7 -DUSE_ZSTD -DCOMPRESSION_THREADS=6 -DREGULAR -o objsr/decompress.o
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -o rtbver objsr/tbver.o objsr/decompress.o objsr/threads.o objsr/checksum.o objsr/city-c.o objsr/util.o -lzstd
make[1]: Leaving directory '/home/oppie/tb/src'
make[1]: Entering directory '/home/oppie/tb/src'
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -c tbverp.c -DMAGIC -DUSE_POPCNT -DTBPIECES=7 -DUSE_ZSTD -DCOMPRESSION_THREADS=6 -DREGULAR -o objsr/tbverp.o
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -o rtbverp objsr/tbverp.o objsr/decompress.o objsr/threads.o objsr/checksum.o objsr/city-c.o objsr/util.o -lzstd
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: note: see the ‘-flto’ option documentation for more information
make[1]: Leaving directory '/home/oppie/tb/src'
make[1]: Entering directory '/home/oppie/tb/src'
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -c tbcheck.c -DMAGIC -DUSE_POPCNT -DTBPIECES=7 -DUSE_ZSTD -DCOMPRESSION_THREADS=6 -DREGULAR -o objsr/tbcheck.o
gcc -pthread -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c11 -Wno-array-bounds -flto -o tbcheck objsr/tbcheck.o objsr/threads.o objsr/checksum.o objsr/city-c.o objsr/util.o
make[1]: Leaving directory '/home/oppie/tb/src'
[oppie@dell src]$

static void decompress(struct CompressState *state, void *dst, sizt_t chunk,
sizt_t has to be size_t i guess

Syzygy tables are compressed using huffman encoding or lzp, if I am not wrong, I was thinking if we can use context mixing compression, like mcm, https://github.com/mathieuchartier/mcm

I think this is alot faster than even 7zip with more compression.

Benchmarks of mcm vs others, http://mattmahoney.net/dc/text.html
and we can see its too fast even at maximum settings from benchmarks, I have personally used it after compiling the source, so thats why suggesting the use of.

So what are your thoughts on this?

New thread for discussing 7-men generation.

Hello,

Could you merge the changes root_probe_wdl (as well as the search.cpp SF example) into this repo ?

Thanks!

The biggest fishtest contributor has a machine with 1tb of ram and is attempting to build 7 piece TB.
https://groups.google.com/d/msg/fishcooking/GgsALC0GOxs/duzznHXHAQAJ

I think I know how to fix the permute indexing that crashed him so far like so.

But I'm not sure about how you found max_ply of 509 and reduce_ply of 122 and reduce_ply_red in rtbgen.c. Is this related to what you said:
"something needs to be done as well to handle very large DTZ values. That only affects a few tables, though. (My idea is to simply divide them by 2 for DTZ > 50. Then everything still fits.)"

Checksum for KRPPvKRP.rtbw is 306b44858171f2141f10edaa7f76ec3a
but in commit it a92493e82147cc425316074c91a0374c

2b86597#diff-8e60e7b5e480d6ac034c260744c8cfdaR294

Which sum is correct?

This would facilitate a wrapper script to pipe errors and so on. See http://google-styleguide.googlecode.com/svn/trunk/shell.xml#STDOUT_vs_STDERR

Hi
I have i7 with 16GB memory, clean 64-bit Ubuntu install
All 5 men tbs generate quickly
I get the following:
rtbgen KQRvKRR -d
Found 145 tablebases.
number of threads = 1
Could not allocate sufficient memory.
Any chance?

On a Raspberry Pi (CPU armv7 , 32 bit OS Linux) , with 6 man syzygy tablebases, and 1GB ram, position fen 2bk4/Pp1pr3/3K3N/2P5/2P5/8/8/8 w - -, we get this error on the 9th ply
(code modified for the following output)
set syzygypath /opt/picochess/tablebases/syzygy
info string Found 510 tablebases
Confirmation: syzygypath set to /opt/picochess/tablebases/syzygy
position fen 2bk4/Pp1pr3/3K3N/2P5/2P5/8/8/8 w - -

go depth 9
info depth 1 seldepth 3 multipv 1 score cp 190 nodes 47 nps 9400 tbhits 0 time 5 pv a7a8q
info depth 2 seldepth 4 multipv 1 score cp 190 nodes 73 nps 14600 tbhits 0 time 5 pv a7a8q e7e6 d6d5 e6h6
info depth 3 seldepth 5 multipv 1 score cp 190 nodes 93 nps 18600 tbhits 0 time 5 pv a7a8q e7e6 d6d5 e6h6 a8b8
info depth 4 seldepth 6 multipv 1 score cp 190 nodes 129 nps 21500 tbhits 0 time 6 pv a7a8q e7e6 d6d5 e6h6 a8b8 h6e6
info depth 5 seldepth 7 multipv 1 score cp 190 nodes 178 nps 29666 tbhits 0 time 6 pv a7a8q e7e6 d6d5 e6h6 a8b8 h6e6 d5d4
info depth 6 seldepth 8 multipv 1 score cp 190 nodes 244 nps 40666 tbhits 0 time 6 pv a7a8q e7e6 d6d5 e6h6 a8b8 h6e6 d5d4 e6e2
info depth 7 seldepth 9 multipv 1 score cp 190 nodes 326 nps 46571 tbhits 0 time 7 pv a7a8q e7e6 d6d5 e6h6 a8b8 h6e6 d5d4 e6e2 b8g3
info depth 8 seldepth 16 multipv 1 score cp 207 nodes 3601 nps 109121 tbhits 1 time 33 pv a7a8q e7e6 d6d5 e6h6 a8b8 h6e6 d5d4 e6c6 b8e5 c6e6 e5g5 d8c7 g5f5 b7b6
Could not mmap() /opt/picochess/tablebases/syzygy/KRBPvKQ.rtbw

using the same tablebases with 64 bit macOS , there is no error.

I believe it may have something to with the number of files being open? Not sure.

Thanks in advanced.

I was able to make the program easily on Linux but when I am trying to use make I get the following output

make[1]: Entering directory 'E:/Syzygy/tb/src'
sh.exe -ec 'gcc -pthread -M -DMAGIC -DUSE_POPCNT -DTBPIECES=7 -DUSE_ZSTD -DCOMPRESSION_THREADS=6 -DREGULAR -msse4.2 -march=corei7 util.c \
        | sed s/util\.o/util\.o\ util\.P/ >.depsr/util.P'
process_begin: CreateProcess(NULL, sh.exe -ec "gcc -pthread -M -DHYPER -DUSE_POPCNT -DTBPIECES=7 -DUSE_ZSTD -DCOMPRESSION_THREADS=6 -DREGULAR -msse4.2 -march=corei7 util.c     | sed s/util\.o/util\.o\ util\.P/ >.depsr/util.P", ...) failed.
make (e=2): The system cannot find the file specified.
Makefile.general:23: .depsr/util.P: No such file or directory
make[1]: *** [Makefile.general:34: .depsr/util.P] Error 2
make[1]: Leaving directory 'E:/Syzygy/tb/src'
make: *** [Makefile:43: rtbgen] Error 2

The compiler is creating files .depsr and -p in the src folder which are empty

Is this code could be improved to generate Syzygy 8 ? I'm really interesting in helping for the coding part bug also provide high end computing capability and storage to achieve that.

It could be useful to update the probing code for the 7men tb

to speed up the reading of the TB, I copy the entire database into RAM (5men), as syzygy does not guarantee the shortest way to victory, I would like to save in the syzygy TB, only the positions that are won with more than 3 moves, making the number a variable, since, if the hardware is more powerful raises the bar, or just to make tests, is this idea reasonable according with database structure?

thanks you!

If you enter this position:
2r1k2r/3R4/2p2K2/8/8/8/8/3R4 w k -
On the chess cloud interface to the 7 man Syzygy files, you get this response:
"This position is not analyzed yet, you can:"
and then there is a [Request for Analysis] button.
If you press the button, it says:
"Too few pieces on the board. You can:"
and then there is a button that says "AI Move"
If you press that button, nothing happens.
The cloud interface:
https://www.chessdb.cn/queryc_en/

It is also possible that there is a problem with the file.

Would like to re-establish contact with Ronald de Man via email if possible.

I'm on the usual email address. Thanks,

Guy Haworth

/usr/include/bits/sigset.h has been removed upstream in the glibc-headers package.

Ubuntu...
https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1720339

Fedora 27....
I don't see it in the latest version.

Error...

make[1]: *** No rule to make target '/usr/include/bits/sigset.h', needed by 'threads.o'. Stop.

It seems that the last commit changed checksums for the tables KPvK KQvK KRvK.
dtz345.txt and wdl345.txt need to be updated.

There are some array overruns in this code that are causing me fits.

1. In rtbgen.c rt and rtbgenp.c in the definition of calc_broken, idx is restricted to be less than end but that doesn't mean that idx+63 is less than end in the inner loop.

2. In the definitions of read_symbol and write_symbol in compress.c, the u8 case basically looks at *&x and *&(x + 1). But if x is at the end of the array (and it sometimes is), this can cause trouble.

The first of these is easy to fix by adjusting the upper limit of the inner loop from 64 to min(64,end-idx). The second is much more confusing.