luajit / luajit-test-cleanup Goto Github PK

This repo is cool! I am really happy to have a test suite. This seems great for people who want to maintain their own branches and keep track of how they compare with everybody else's. Like, have I broken something? Have my optimizations worked? Has somebody else made some optimizations that I should merge? etc. Just now I would like to maintain a branch called lowlevel to soak up things like intrinsics and DynASM Lua-mode so this is right on target for me.

I whipped up a Continuous Integration job to help. The CI downloads the latest code for some well-known branches, runs the benchmark suite 100 times for each branch, and reports the results. This updates automatically when any of the branches change (including the benchmark definitions).

The reason I run the benchmarks 100 times is to support tests that use randomness to exercise non-determinism in the JIT, like roulette (#9). Repeated tests mean that we can quantify how consistent the benchmark results are between runs, and once we have a metric for consistency then it is more straightforward to optimize (see LuaJIT/LuaJIT#218).

The branches I am testing now are master, v2.1, agentzh-v2.1, corsix/x64, and lukego/lowlevel. If anybody would like a branch added (or removed) just drop me a comment here. Currently the benchmark definitions are coming from my fork because I wanted to include roulette to check that variation is measured correctly.

Screenshot of the first graph (click to zoom):

and links:

• Current results at the time of writing.
• Permalink to the latest results.
• CI Jobset page where all builds can be found (and also related files like the raw CSV.
• Definition of the test runner written in Nix + shell.
• Rmarkdown source for the report.

Hope somebody else finds this useful, too! Feedback & pull requests welcome. I plan to keep this operational.

Split up the tests in to many small tests, and convert them into the format required by the testing framework where applicable. Reorganize them into the new directory structure.

This issue will track progress for adding the test suites from Lua (http://www.lua.org/tests/) to this repository. Modifications will be made to the test suites where necessary to make them work with LuaJIT.

It maybe possible to backport additional tests from 5.2.x and 5.3.x series where the language & libraries are compatible.

I will do the work on a branch initially and once working will merge it to the master.

License: The Lua Test suites are under MIT license.

Hello.

$ luajit test.lua 219
[219/492] lib/string/format/num.lua --- ExploringBinary.com/print-precision-of-dyadic-fractions-varies-by-language/
lib/string/format/num.lua:13: expected string.format("%.99e", "0") == "4.940656458412465441765687928682213723650598026143247644255856825006755072702087518652998363616359924e-324", but got "0.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e+00"
stack traceback:
[C]: in function 'error'
lib/string/format/num.lua:13: in function 'check'
lib/string/format/num.lua:163: in function <lib/string/format/num.lua:160>
[C]: in function 'xpcall'
test.lua:378: in function 'execute_plan'
test.lua:413: in main chunk
[C]: at 0x55fc2171a080
0 passed, 1 failed

x86_64 system, luajit revision: e296f56b825c688c3530a981dc6b495d972f3d01

Test 366 is failing for me with the v2.1 branch (commit LuaJIT/LuaJIT@d3e36e7):

$ ~/git/luajit/src/luajit test.lua 366
[366/508] lib/contents.lua --- pre-5.2 table
lib/contents.lua:80: got: "concat:foreach:foreachi:getn:insert:maxn:move:remove:sort"
expected: "concat:foreach:foreachi:getn:insert:maxn:remove:sort"
stack traceback:
        [C]: in function 'error'
        lib/contents.lua:17: in function 'check'
        lib/contents.lua:80: in function <lib/contents.lua:79>
        [C]: in function 'xpcall'
        test.lua:378: in function 'execute_plan'
        test.lua:413: in main chunk
        [C]: at 0x00404a70
0 passed, 1 failed

Any tips on how to resolve this?

I suggest we use a declarative data-driven format to organize the test cases in each individual test file so that it is independent of the actual test runner and framework. A good test specification syntax I've been using myself for many years is TestML:

http://testml.org/specification/language//index.html

This way we can free ourselves in using what way or what combinations of ways to run the tests without touching the test files themselves.

This also makes it easy when we add unit tests for LuaJIT's internal pipelines (like individual JIT compiler's optimization passes and intermediate IR form) in the future.

I hope we can draw a clean line between test case representation and test running strategies.

ubuntu@arm:~$ ./sandbox/orig/LuaJIT/src/luajit LuaJIT-test-cleanup/test/test.lua 220
[220/508] lib/string/format/num.lua --- ExploringBinary.com/print-precision-of-dyadic-fractions-varies-by-language/
LuaJIT-test-cleanup/test/lib/string/format/num.lua:13: expected string.format("%.99e", "0") == "4.940656458412465441765687928682213723650598026143247644255856825006755072702087518652998363616359924e-324", but got "0.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e+00"
stack traceback:
[C]: in function 'error'
LuaJIT-test-cleanup/test/lib/string/format/num.lua:13: in function 'check'
LuaJIT-test-cleanup/test/lib/string/format/num.lua:163: in function <LuaJIT-test-cleanup/test/lib/string/format/num.lua:160>
[C]: in function 'xpcall'
LuaJIT-test-cleanup/test/test.lua:378: in function 'execute_plan'
LuaJIT-test-cleanup/test/test.lua:413: in main chunk
[C]: at 0x00469c01
0 passed, 1 failed

ARM system, luajit revision: 1d7b5029c5ba36870d25c67524034d452b761d27

Devise a logical naming scheme and hierarchy / directory structure for the tests.

Howdy! I am a first-time contributor with PR #9 and I have some questions:

• Who is responsible for deciding when to merge my code?
• What are this person's requirements?
• What actions do I need to take to satisfy them?

Anyone who wants to help with the cleanup effort and feels qualified, please apply here for commit access to this repo.

Note: this does not automatically grant you commit access to the LuaJIT main repo, which is a tad more sensitive.

Agree on a testing library and a test runner with minimal dependencies, preferably Lua/shell only.
Test cases should still be easy to run without any (big) dependencies.