Giter VIP home page Giter VIP logo

fsharp-hedgehog-experimental's Introduction

fsharp-hedgehog-experimental

NuGet AppVeyor

Hedgehog with batteries included: Auto-generators, extra combinators, and more.

Features

  • Auto-generation of arbitrary types
  • Generation of functions
  • Lots of convenient combinators

Examples

Convenience combinators

Generate lists without having to explicitly use Range:

let! exponentialList = Gen.bool |> GenX.eList 1 5 // Same as Gen.list (Range.exponential 1 5)
let! linearList      = Gen.bool |> GenX.lList 1 5 // Same as Gen.list (Range.linear 1 5)
let! constantList    = Gen.bool |> GenX.cList 1 5 // Same as Gen.list (Range.constant 1 5)

Generate strings without having to explicitly use Range:

let! exponentialStr = Gen.alpha |> GenX.eString 1 5 // Same as Gen.string (Range.exponential 1 5)
let! linearStr      = Gen.alpha |> GenX.lString 1 5 // Same as Gen.string (Range.linear 1 5)
let! constantStr    = Gen.alpha |> GenX.cString 1 5 // Same as Gen.string (Range.constant 1 5)

Generate a random shuffle/permutation of a list:

let lst = [1; 2; 3; 4; 5]
let! shuffled = GenX.shuffle lst // e.g. [2; 1; 5; 3; 4]

(Note that the shuffle may produce an identical list; this is more likely for shorter lists.)

Shuffle/permute the case of a string:

let str = "abcde"
let! shuffled = GenX.shuffleCase str // e.g. "aBCdE"

Generate an element that is not equal to another element (direct or option-wrapped):

// Generates an int that is not 0
let! notZero = Gen.int (Range.exponentialBounded()) |> GenX.notEqualTo 0

// Also generates an int that is not 0
let! notZero = Gen.int (Range.exponentialBounded()) |> GenX.notEqualToOpt (Some 0)

// Can generate any int
let! anyInt = Gen.int (Range.exponentialBounded()) |> GenX.notEqualToOpt None

Generate a string that does not equal another string, start with another string, or is a substring of another string:

let strGen = Gen.alpha |> GenX.lString 1 5

let! ex1 = strGen |> GenX.notEqualTo "A" // Does not generate "A" (same function as previous example)
let! ex2 = strGen |> GenX.iNotEqualTo "A" // Case insensitive, does not generate "A" or "a"
let! ex3 = strGen |> GenX.notSubstringOf "fooBar" // Does not generate e.g. "Bar" (but "bar" is OK)
let! ex4 = strGen |> GenX.iNotSubstringOf "fooBar" // Case insensitive, does not generate e.g. "Bar" or bar"
let! ex5 = strGen |> GenX.notStartsWith "foo" // Does not generate e.g. "foobar" (but "Foobar" is OK)
let! ex6 = strGen |> GenX.iNotStartsWith "foo" // Case insensitive, does not generate e.g. "foobar" or "Foobar"

Generate an item that is not in a specified list:

let! str = Gen.int (Range.exponentialBounded()) |> GenX.notIn [1; 2] // Does not generate 1 or 2

Generate a list that does not contain a specified item:

// Produces a list not containins 2, e.g. [1; 5; 7]. Note that this is a filter and not
// a removal after the list has been generated, so the length of the list is unaffected.
let! intList = Gen.int (Range.exponentialBounded()) |> GenX.cList 3 3 |> GenX.notContains 2

Generate a list that contains a specified item at a random index:

// Generates a list and inserts the element. The list is 1 element longer than it would otherwise have been.
let! intList = Gen.int (Range.exponentialBounded()) |> GenX.cList 3 3 |> GenX.addElement 2

Generate null some of the time, or don't generate nulls:

let strGen = Gen.alpha |> GenX.eString 1 5

// Generates null part of the time (same frequency as Gen.option generates None)
let nullStrGen = strGen |> GenX.withNull

// Does not generate null
let noNullStrGen = nullStrGen |> GenX.noNull

Generate sorted/distinct tuples (2, 3 or 4 elements):

// Can produce 'a', 'a', 'c' but not 'a', 'c', 'b'
let! x, y, z = Gen.alpha |> Gen.tuple3 |> GenX.sorted3

// Can produce 'b', 'a', 'c' but not 'b', 'b', 'c'
let! x, y, z = Gen.alpha |> Gen.tuple3 |> GenX.distinct3

// Strictly increasing - can produce 'a', 'b', 'c' but not 'a', 'a', 'c'
let! x, y, z = Gen.alpha |> Gen.tuple3 |> GenX.increasing3

Generate a date range:

// Generates two dates at least 1 and at most 10 days apart, each with random time of day.
// The interval increases linearly with the implicit size parameter.
let! d1, d2 = GenX.dateInterval (Range.linear 1 10)

Generate a function:

// Generates a list using inpGen together with a function that maps each of the distinct
// elements in the list to values generated by outGen. Distinct elements in the input list
// may map to the same output values. For example, [2; 3; 2] may map to ['A'; 'B'; 'A'] or
// ['A'; 'A'; 'A'], but never ['A'; 'B'; 'C']. The generated function throws if called with
// values not present in the input list.
let intGen = Gen.int (Range.exponentialBounded())
let charListGen = Gen.alpha |> GenX.eList 1 10
let! chars, f = charListGen |> GenX.withMapTo intGen
// chars : char list
// f : char -> int


// Generates a list using inpGen together with a function that maps each of the
// distinct elements in the list to values generated by outGen. Distinct elements
// in the input list are guaranteed to map to distinct output values. For example,
// [2; 3; 2] may map to ['A'; 'B'; 'A'], but never ['A'; 'A'; 'A'] or ['A'; 'B'; 'C'].
// Only use this if the output space is large enough that the required number of distinct
// output values are likely to be generated. The generated function throws if called with
// values not present in the input list.
let intGen = Gen.int (Range.exponentialBounded())
let charListGen = Gen.alpha |> GenX.eList 1 10
let! chars, f = charListGen |> GenX.withDistinctMapTo intGen
// chars : char list
// f : char -> int

Auto-generation

Generate any type automatically using default auto-generators for primitive types:

// Can generate all F# types (unions, records, lists, etc.) as well as POCOs
// with mutable properties or constructors.

type Union =
  | Husband of int
  | Wife of string
  
type Record =
  {Sport: string
   Time: TimeSpan}
   
// Explicit type parameter may not be necessary if it can be inferred.
let! union = GenX.auto<Union>
let! record = GenX.auto<Record>

// Recursive types are supported. By default, recurses at most once (subject to change).
type Recursive =
  {OptChild: Recursive option
   LstChild: Recursive list}
let! recursive = GenX.auto<Recursive>
// E.g. {OptChild = Some {OptChild = None; LstChild = []}; LstChild = []}
// Note that you may need to adjust the defaults when any kind of sequence is involved
// (see below), since by default the range for generated sequences are Range.exponential 0 50
// (subject to change).

Generate any type automatically and override default generators and settings:

let! myVal =
  {GenX.defaults with 
     SeqRange = Range.exponential 1 10
     RecursionDepth = 2}
  // Will use this generator for all ints
  |> AutoGenConfig.addGenerator (Gen.int (Range.linear 0 10))
  // Will use this generator when generating its return type
  |> AutoGenConfig.addGenerator Gen.myCustomGen
  // Generate using the config above
  |> GenX.autoWith<MyType>

If you’re not happy with the auto-gen defaults, you can of course create your own generator that calls GenX.autoWith with your chosen config and use that everywhere.

Deployment checklist

For maintainers.

  • Make necessary changes to the code
  • Update the changelog
  • Update the version and release notes in the fsproj file (incrementing the version on master is what triggers the deployment to NuGet)
  • Commit and push

Each commit to master

fsharp-hedgehog-experimental's People

Contributors

alexeyraga avatar cmeeren avatar dharmaturtle avatar marklam avatar moodmosaic avatar tysonmn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

fsharp-hedgehog-experimental's Issues

Must AutoGenConfig be a struct?

In commit b52c847, I added [<CLIMutable>] and [<Struct>] to AutoGenConfig, with the motivation to make it easier to use from C#. I can understand how CLIMutable achieves that goal, but I can't figure out why Struct should make it easier.

Furthermore, since AutoGenConfig has so many fields, having it be a Struct seems like it could be bad for performance (I have not measured).

Could anyone illuminate me? Is Struct relevant for C# interop?

Should we transfer this repo to hedgehogqa?

It may help people discover it more easily! I think anyone using fsharp-hedgehog at least ought to know about this repo and its auto gen capability.

The only 'requirement' is that at least @jacobstanley and/or myself would need to have push access. But don't let that scare you, we work via pull requests all the time.

Remove unit parameter from auto

Since auto<'a>() has an explicit type parameter, I think it's possible to remove the unit parameter (if it's possible, then IMHO we should do it since it's just unnecessary noise). It compiled fine when I tried it, and all tests completed successfully.

Waiting with PR until after #21 is merged due to conflicts.

Transfer ownership to @cmeeren

Given that

  • @cmeeren has done most (if not all) of the hard work on fsharp-hedgehog-experimental
  • my spare time is extremely limited when it comes to maintaining fsharp-hedgehog-experimental

I suggest that we

  • transfer fsharp-hedgehog-experimental to @cmeeren πŸŽ‰ πŸš€

Advantages

Transferring this repository to @cmeeren can make it move a little bit faster, as right now the issues and the pull requests can be blocked several weeks waiting feedback from us.

Of course, I'm more than happy to help with any questions on issues and pull requests (given that I have the answer(s)), but I will no longer be the maintainer of this repository.

By not being the maintainer of fsharp-hedgehog-experimental, I may put my (limited) spare time into fsharp-hedgehog itself and even the Haskell version (where we're missing the diehard tests, for example).

Disadvantages

Are there any?

Practical issues

I'd really like this repository to be moved to @cmeeren's GitHub, (mine or some other's, if @cmeeren isn't willing to take over).

Moving fsharp-hedgehog-experimental outside @hedgehogqa amplifies that fact that this repository is all about taking great ideas from the community and grouping them altogether for trying them out, like:


I've discussed this with @jystic and he's positive, so we really look forward your decision @cmeeren πŸ₯‡

Auto-gen implementation

Making a PR of the auto-generator now. Where do you want it placed? IMHO it should be accessible as Hedgehog.Gen.auto, but the implementation is fairly substantial, so I'm wondering if it should be placed in its own file (and if so, how to do that considering that Hedgehog.Gen is already defined in Gen.fs) or if it's okay to have everything in Gen.fs (which might be the only option).

Add ability to override custom types in auto-generator

It would be great if the user could specify overrides for any complex type they desire. For example, say Customer is a type that has a field of type Customer list (sub-customers of the current customer). This is recursive and (currently) needs a specifically designed generator to avoid stack overflow. Say then that I want to generate an Order type which, among other things, contains a Customer, then I'd want to just say auto<Order>(...) and somehow specify that I want Customer objects to be generated using my specific generator, and otherwise the defaults are fine.

I'm not sure how to do this though, or if it is easy/feasible/possible.

Adding useful types to AutoGenConfig

Similarly to what FSCheck is doing

Can we consider adding types like:

NegativeInt
NonNegativeInt
PositiveInt
NonZeroInt
NormalFloat
NonEmptyString
StringNoNulls
NonWhiteSpaceString
XmlEncodedString
UnicodeChar
UnicodeString
Interval
IntWithMinMax
NonEmptySet
NonEmptyArray
FixedLengthArray

to AutoGenConfig so that they are available by default?

I know that they can be defined separately and configured via Property attribute, but since they are very generic and very useful having them "out-of-the-box" would be extremely convenient...

Support immutable collections

I am using collections from System.Collections.Immutable (ImmutableDictionary in my case) and am getting the following exception:

System.Exception: Class System.Collections.Immutable.ImmutableDictionary`2[System.String,System.String] lacking an appropriate ctor

System.Exception
Class System.Collections.Immutable.ImmutableDictionary`2[System.String,System.String] lacking an appropriate ctor
   at Microsoft.FSharp.Core.PrintfModule.PrintFormatToStringThenFail@1433.Invoke(String message) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\printf.fs:line 1433
   at Hedgehog.GenX.mkRandomMember@397-2.TypeShape-Core-Core-IMemberVisitor`2-Visit[b](ShapeMember`2 shape) in /home/runner/work/fsharp-hedgehog-experimental/fsharp-hedgehog-experimental/src/Hedgehog.Experimental/Gen.fs:line 400
   at Hedgehog.GenX.autoInner[a](AutoGenConfig config, FSharpMap`2 recursionDepths)
   at Hedgehog.GenX.autoInner@525-40.TypeShape-Core-Core-IConstructorVisitor`2-Visit[CtorParams](ShapeConstructor`2 ctor) in /home/runner/work/fsharp-hedgehog-experimental/fsharp-hedgehog-experimental/src/Hedgehog.Experimental/Gen.fs:line 528
   at InternalLogic.genxAutoBoxWith[T](AutoGenConfig x)

which points to here

Since immutable collections are part of BCL, it would be beneficial to support them.
Would it be possible to do?

Update NuGet API key

It has been a while since I have been a maintainer of this package, but I just noticed that NuGet publishes are still done in my name, presumably because it's using my API key:

image

@TysonMN, I understand you to be the current primary maintainer of this package. I have just added you as an owner of the NuGet package. Whenever it's convenient for you, consider creating and using your own API key and removing me as a NuGet package owner.

.NET `List<T>` isn't generated properly

Using HH from C# and this problem arises:

[Property]
public void SomeTest(List<int> values) { ... }

Then running this test fails with this exception:

System.ArgumentOutOfRangeException
capacity was less than the current size. (Parameter 'value')
   at System.Collections.Generic.List`1.set_Capacity(Int32 value)

Perhaps a bug in a List generator?

Limit recursion depth based on size instead of configuration value

PR #21 added a configuration value called RecursionDepth to prevent stack overflows when generating a recursive type (c.f. #19 (comment)).

FsCheck solves this problem by relating the recursion depth to the size parameter.

However, a recursive generator like this may fail to terminate with a StackOverflowException, or produce very large results. To avoid this, recursive generators should always use the size control mechanism

This seems like a better solution to me, but I am just getting seriously into property-based testing, so I am probably unaware of all the considerations.

What are the tradeoffs for this configuration value vs utilizing the size parameter?

Add permutation generators

It'd be nice with generators that permute existing sequences/lists/strings:

/// Generates a permutation the specified list (shuffles its elements).
permutationOf (list : 'a list) : Gen<'a list>

/// Randomizes the case of the characters in the string.
casePermutationOf (str : string) : Gen<string> =

See the combinators removed in this commit. An open question is how to handle the randomness generation - System.Random might not be the correct choice.

GenX.auto fails on TimeSpan

module HedgehogExperimentalRepro.Tests

open System
open Hedgehog
open FsUnitTyped

[<Test>]
let ``can make timespan`` () =
    property {
        let! (test : TimeSpan) = GenX.auto
        test.Seconds |> shouldBeSmallerThan 100
    } |> Property.check

fails with the exception

System.OverflowException : Value was either too large or too small for an Int64.
at System.Numerics.BigInteger.op_Explicit(BigInteger value)
   at <StartupCode$Hedgehog>[email protected](BigInteger x)
   at [email protected](Int32 sz)
   at [email protected](Seed seed, Int32 size)
   at [email protected](Seed seed, Int32 size)
   at [email protected](Seed seed0, Int32 size)
   at [email protected](Seed seed0, Int32 size)
   at Hedgehog.Property.loop@332-21(Int32 n, Random`1 random, Seed seed, Int32 size, Int32 tests, Int32 discards)
   at Hedgehog.Property.Report(Int32 n, Property`1 p)
   at Hedgehog.Property.Check(Property`1 p)
   at HedgehogExperimentalRepro.Tests.can make timespan() 

A bug when handling inheritance with `new` members

Hi, I have found one of the edge cases in which GenX fails to generate a correct instance, as well as the possible solution to the issue.

Problem

Assuming that we have a hierarchy of two types:

public abstract class A { public object Value { get; protected set; } }
public class B : A { public new string Value { get { return (string)base.Value; } set { base.Value = value; } } }

Notice that B has a property with the same name as A, but "hides" the base property with the new keyword.

Now, when we ask GenX.auto<B> then it generates an incorrect instance because Value property gets obj as a value and not string.

Why it is happening

It is happening because this code:

| Shape.CliMutable (:? ShapeCliMutable<'a> as shape) ->
    let props = shape.Properties |> Seq.toList

gives us the list of two properties in this order:

  1. Value that is declared in B
  2. Value that is declared in A

and values for these properties are generated and set in this order.
So first, GenX will generate a correct (by intention) value and will set it to B.Value, but then it will generate an empty object and pass it into the "hidden" property, which overwrites the correct value with an incorrect one.

The list of properties comes to us from TypeShape, which just reflects all the instance properties in a given type:

and [<Sealed>] ShapeCliMutable<'Record> private (defaultCtor : ConstructorInfo) =
    let properties = lazy(
        typeof<'Record>.GetProperties(AllInstanceMembers)
        |> Seq.filter (fun p -> p.CanRead && p.CanWrite && p.GetIndexParameters().Length = 0)
        |> Seq.map (fun p -> mkWriteMemberUntyped<'Record> p.Name p [|p|])
        |> Seq.toArray)

GetProperties method explicitly does not specify the order in which properties are returned:

The GetProperties method does not return properties in a particular order, such as alphabetical or declaration order. Your code must not depend on the order in which properties are returned, because that order varies.

so that we cannot rely on "this instance" property being passed last (and it does, indeed, not happen).

Possible solution

Since we cannot rely on the order, one possible way of working around the issue would be to make sure that properties of "base instance" are set first, and properties of "this instance" are set last.

This would achieve that:

| Shape.CliMutable (:? ShapeCliMutable<'a> as shape) ->
    let selfProps, baseProps =
      shape.Properties
      |> Seq.toList
      |> List.partition (fun x -> x.MemberInfo.DeclaringType = typeof<'a>)

    (baseProps @ selfProps)  // <---- here is the change
    |> Seq.toList
    |> ListGen.traverse memberSetterGenerator
    |> Gen.map (fun fs -> fs |> List.fold (|>) (shape.CreateUninitialized ()))

and this way GenX will produce "correct by intention" value.

Of course, a "better" solution would be to detect new and to remove "hidden" members from the list, but:

  1. I wasn't able to figure out how to do that reliably :)
  2. Because types that we generated are blackboxes, it may be logical to still initialise the base values with something, as we don't (and can't) know whether the child type getter/setter are accessing the base one or not.

So I think that the solution with setting "this instance" properties first should be good enough, and should definitely be better to what we currently have.

Question

Will you accept a PR that fixes this issue in this way? ;)

Add support to Nullable<T>

Do you think that support for Nullable<'T> can be added, similarly to how Option<'T> is now supported?
It is useful for property-testing with types that are defined in C#.

Add reference to Hedgehog

There needs to be a reference to the main Hedgehog library. I don't know what kind of constraints you want on the version number, so I'll leave it to you.

Support for Array.Rank greater than 1

Why isn't there support when Array.Rank is greater than 1? Is there some fundamental technical difficulty or is it just that sufficient time/effort was never spent to implement that case?

| Shape.Array s when s.Rank = 1 ->
s.Element.Accept {
new ITypeVisitor<Gen<'a>> with
member __.Visit<'a> () =
if canRecurse typeof<'a> then
autoInner<'a> config (incrementRecursionDepth typeof<'a>) |> Gen.array config.SeqRange |> wrap
else
Gen.constant ([||]: 'a array) |> wrap}
| Shape.Array _ ->
raise (NotSupportedException("Can only generate arrays of rank 1"))

DateTimeOffset auto-generator always generates zero offset

Just noticed (from static reading of the code) that the auto-generator seems to always generate DateTimeOffset values with zero offset:

|> AutoGenConfig.addGenerator (Gen.dateTime dateTimeRange |> Gen.map DateTimeOffset)

If that is the case, it is probably a good idea to generate various offsets. Perhaps using Gen.dateTimeOffset?

This is not an important issue for me, and I can't prioritize making a PR right now, but I wanted to mention it since I saw it.

Does Hedgehog return an incorrect value on failure?

I am running a test with a simple condition:

match message with
| Happy s when s.Reason.Contains('a') -> Nack
| _ -> Ack

I see in debug that it fails properly and the value is shrunk properly to the failure case of a single a character (see the screenshot)

image

I also see in debug that this is the last shrink, i.e. the test is not getting called anymore.

However, the result of the test is printed as:

*** Failed! Falsifiable (after 68 tests and 6 shrinks):
[Happy { "reason": "`", "level": -287093 }]

Or see this screenshot
image

Which is not a correct failing condition, it should say "reason": "a" instead!

Interestingly, I also have a numeric field in that structure, and when I do something like

| Happy s when s.Level = 97 -> Nack

Then the failure is reported correctly. So it looks like it is working for numbers, but not for strings

Split (or merge) the library?

It looks like the library consists of two parts:

  1. A set of useful extensions for Hedgehog. These are all these sorted, noNull, shuffle, uri generators and operations on them
  2. The auto-generating part (heavy reflection, AutoGenConfig, GenX.auto, etc.)

These parts do not seem to depend on each other.

The questions are:

  • Should we create another library for useful Gen/Range combinators and constructors? Some of them may even go to HH itself.
  • Is the autogen part still "experimental"? I understand why we may not want to put it into HH itself. But maybe it is stable/useful enough so that something like Hedgehog.Autogen can be born as its own library?

GenX.auto fails with anything involving uint64

e.g.

let! foo = GenX.auto<uint64>

Fails with

System.Exception : Class System.UInt64 lacking an appropriate ctor
   at Microsoft.FSharp.Core.PrintfModule.PrintFormatToStringThenFail@1647.Invoke(String message)
…
   at [email protected](Unit x)
   at [email protected](Unit x)
   at [email protected](Seed seed, Int32 size)
   at Hedgehog.Property.loop@332-21(Int32 n, Random`1 random, Seed seed, Int32 size, Int32 tests, Int32 discards)
   at Hedgehog.Property.Report(Int32 n, Property`1 p)
   at Hedgehog.Property.Check(Property`1 p)

This also affects discriminated unions with uint64 cases.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.