dalekbaldwin / check-it Goto Github PK

Using a manually-expanded DSL means we can re-use built-in Lisp symbols like list and integer, which is more readable than having to name everything list-gen or integer-gen.

But this means we can't get SLIME to give usage hints in the mini-buffer the way we would if each generator type were constructed with an explicit function or macro.

I want to keep the DSL simple enough so that won't bother anybody learning how to use it. Please let me know if it's a problem.

Hi,

I think it would be better if the string generator would generate whole ascii table strings. What I mean is that those strings should contain special characters like $#%&* etc. Unprintable characters are debatable, but I see some use cases, too.

Let me here what you think and how you feel about the unprintable characters corner cases.

Best,
Sebastian

Streaming generators would return a function that the test body can call to get as many times as it needs to get the values as it needs.

https://hypothesis.readthedocs.org/en/latest/data.html#infinite-streams

The generation history could be stored as a list paired with a function that iteratively produces values from it so that the list could be shrunk/saved as a regression case.

These would probably only make sense as top-level generators, not as subgenerators of some other generator.

Now that custom generators don't rely on backquoted templates, that functionality can be separated into custom macro-like patterns.

*size*, *list-size*, etc. can still serve as hard bounds, but for each such parameter, another parameter could start small and gradually approach the value of *size*/*list-size*. There could also be an option to measure the time it takes to run each test and stop increasing the bounds once the iterations start taking too long.

Regression cases could be serialized along with metadata, e.g. the date/time a failure was first detected in order to leave a paper trail documenting where bugs were introduced in a project's version control history.

Number generators (and possibly list lengths) should allow more sophisticated options than a simple uniform distribution. Two ways this might be done: distributions could be added on a case-by-case basis and given a hard-coded meaning in the DSL syntax like (real :distribution (gaussian :mean 5.0 :variance 2.0)), or you could do something like (integer :distribution #'distribution-function) with common cases bundled along with check-it in a utility package, plus building blocks to make it easy to write your own in the way check-it expects.

Also, in an or generator, you should be able to specify custom set of starting biases instead of the default 1.0 for each choice. This would be simple enough to implement, but I'm undecided on the syntax.

One possibility would be (or ((integer) :bias 1.2) ((real) :bias 2.9)); when the head of a sub-typespec is a list, it would be taken as the actual typespec and then whatever follows are options describing how it is managed by the parent generator. At this point I don't know if any other kinds of options would ever be useful though.

This is easier than manually adding them to the regression file and helps to self-document what the values produced by a complicated generator are supposed to look like.

Could be as simple as (check-it (generator (integer)) test :examples (list 5 12)).

Right now all parameters to a custom generator are required, but optional parameters would be helpful -- they would allow something like the character generator to be reimplemented with def-generator.

transparent-wrap's parser assumes lamba lists have already been validated by the Lisp implementation, but def-generator lambda lists will need to be validated before the new generator type is constructed. It may be possible to save the lambda list form, build a do-nothing function with it, and then if Lisp doesn't signal an error while building that function, finally parse the saved lambda list with transparent-wrap's parser to get the variable/slot names. This would lead to style-warnings for unused arguments in the do-nothing function though.

So the right way to go is probably just to steal a good validating lambda list parser from somewhere else.

This generator doesn't work:

(generator
(chain ((n (integer 3 18)))
(generator (tuple
(list (integer 1 75) :length n)
(list (integer 0 (1- n)) :length (* 2 n))))))

generators.lisp calls random with 0 argument
211: (random (- (min (1+ max-length) list-size) min-length)))))

Argument is neither a positive integer nor a positive float: 0
[Condition of type SIMPLE-TYPE-ERROR]

Hi,

I was playing around and tried to define my custom generators. I came up with

(defgenerator numbers () 
  `(generate (generator (or (integer 0 10) 
                            (integer 20 30)))))

to generate integers between [0, 10) and [20, 30). Which worked.

(let ((*size* 50))
            (generate (generator (numbers))))
;;=> 22

Then, I tried to generate a list of numbers like so

(let ((*size* 50))
            (generate (generator (list (numbers)))))
;;=> (7 7 7 7 7 7 7 7 7 7 7 7 7 7)

This didn't met my assumption. What do I need to do to generate a list of random numbers (laying in one of the two intervals above).

Thanks.
Sebastian

Reference: http://stackoverflow.com/questions/30030304/in-allegro-cl-and-abcl-can-i-construct-a-struct-given-only-its-type-name

Also, if a struct's slots are declared with (Lisp) type specifications, check-it should coerce sub-generators' values to fit them.

Hi.

Thanks for writing this nice library. I'm using it to generate random data from schemas: https://mmontone.github.io/schemata/#Data-generation

I'm missing some generators, and I thought I'd share my views.

One, the possibility of lifting a function (including a lambda) as generator.

https://github.com/mmontone/schemata/blob/3ed6f36cbc227ccf69bb0d62d3037ed325d7a2ed/generators.lisp#L16C1-L30

This is my current implementation:

 (defclass func (check-it::custom-generator)
    ((check-it::bias :initform 1.0
                     :accessor check-it::bias
                     :allocation :class)
     (func :initarg :func)))
  (setf (get 'func 'check-it::genex-type) 'generator)
  (setf (get 'func 'check-it::generator-form)
        `(lambda ,'(func) (make-instance ','func ,@'(:func func))))

  (defmethod generate ((generator func))
    (let ((func (slot-value generator 'func)))
      (funcall func)))
  )

and this is how you can use: (generator (func (lambda () (1+ (random 100))))).

I find this generator extremely valuable as it lets me plug arbitrary in-line functions that return random data easily.

This implementation doesn't work because it caches the result:

(def-generator func (func)
  (generator (funcall func)))

But this alternative implementation does work, and is simpler and more direct:

(defmethod check-it::generate ((generator function))
  (funcall generator))

The other generator, is a "member-of" generator, that chooses a random element from a collection (list, hash-table, or other).

This is my current implementation:

https://github.com/mmontone/schemata/blob/3ed6f36cbc227ccf69bb0d62d3037ed325d7a2ed/generators.lisp#L37-L48

I'm using that one quite a lot of too.

I'm using like:

(defparameter *person-names*
  (list "John" "Peter" "Stan" "Steve" "Wendy" "Caroline"))

(def-generator person-name-generator ()
  (generator (member-of *person-names*)))

(def-generator age-generator ()
  (generator (func (lambda () (1+ (random 100))))))

(defschema person
    (object person
            ((name string :generator (generator (person-name-generator)))
             (age integer :required nil
                          :generator (generator (age-generator))))))

So, it would be nice that some implementation of these generators be included as part of check-it, in my opinion.

Why is the range only [-10,10]? For example this will stack overflow: (generate (generator (guard (lambda (x) (> x 10)) (integer))))

There are a bunch of test frameworks out there with their own peculiar interfaces for declaring e.g. that a test case is expected to signal a particular error. There's no reason to attempt a half-assed duplication of those kinds of features with their own idiosyncratic API that people have to learn just to add a few Quickcheck-style tests to their suite using similar features. I'd like the usage pattern to stay as it is, so you can just include a check-it form within a test created with whatever framework you already know and like.

However, generating code for a deterministic test with :gen-output-template and writing it to a file is kinda sketchy. It requires you to redundantly write a testing-framework-level test form within a testing-framework-level test form, and it could probably break in weird ways under unforeseen circumstances. It would be better just to print a readable representation of the data that caused a test to fail and tag it with an ID corresponding to the failed test. But that would require the check-it macro to detect what sort of context it's being used in, so it could associate the name of the test with the correct piece of serialized data to replicate an old test case. It would need to know something about the top-level test definition form surrounding it, or else it would need an extra symbol as a parameter to tag its failure cases.

I think the simplest way to combine test framework integration and sane failure data serialization is to create specific wrappers for popular test frameworks' existing test-defining macros. So if you wanted to make use of a check-it form, you would define the test with deftest-with-check-it instead of deftest. These wrappers would encode the name of the test in a place the check-it macro knows where to look for. The check-it macro would then expand into a form that looks up the existing failure cases, runs them, and then runs new randomized cases.

The failure output file and package could then be declared in a single location, possibly in another wrapper macro surrounding defsuite (or the corresponding macro in whatever test framework).