pfdietz / ansi-test Goto Github PK

The loading scheme for ansi-test has been problematic. I had designed it before asdf, and wanted it to be useful in lisps without asdf. However, I am mostly using it now in lisps that do support asdf. So: add asd files instead of the hacked loading system that's currently there. Separate out rtp and the random tester from other tests. Ultimately, I want to separate the random tester entirely.

The random type prop (rtp) tests encode information about what types are allowed as arguments to built-in functions. This information could be pulled out into a more declarative form, so it could be used elsewhere (for example, when generating random inputs to the compiler it could be used to generate more type-correct code.)

The random tester currently operates by requiring that the generated lisp always be valid (that is, the types of arguments always be permissible when executed). This makes reduction of the forms difficult, as this has to be maintained.

Instead, add a facility where violations can be detected dynamically, so reduced test cases can be seen to be invalid and the reduction discarded. Do this by walking the form and replacing functions with 'safe' versions that manually check constraints and signal an invalid test error if they are violated.

Also, check walk the form and check other constraints. The goal is to make test case reduction be very dumb and aggressive, with bad reductions caught by the checker.

The mutation tester has a reader facility that slurps in code from random lisp files. It has to work even with packages that aren't defined (since it's just generating raw material for stuff that looks vaguely like valid lisp). Previously I hacked SBCL to have a parameter that caused it to not throw errors on bad packages, but this is not portable.

Instead, use Eclector and customize it to do something useful on package errors.