hedgehogqa / fsharp-hedgehog-xunit Goto Github PK

This runs, but passes (or perhaps doesn't run at all?) even though it should fail:

[Hedgehog.Xunit.Property]
public Hedgehog.Linq.Property CanUseSelectWithAssertion()
{
    var property =
        from x in ForAll(Gen.Bool)
        select Assert.True(!x);

    return property; // Assume Hedgehog.Xunit will then run the property(?)
}

If I just use the core library, it runs, and fails, as it should:

[Fact]
public void CanUseSelectWithAssertion()
{
    var property =
        from x in ForAll(Gen.Bool)
        select Assert.True(!x);

    property.Check();
}

Both snippets require the following using directives:

using Xunit;
using Hedgehog.Linq;
using static Hedgehog.Linq.Property;

@dharmaturtle, with @cmeeren we've discussed function generation in the past. Could that work with Hedgehog.Xunit?

using Hedgehog;
using Hedgehog.Xunit;

using System;
using System.Collections.Generic;
using System.Linq;

using Xunit;
using Xunit.Abstractions;

public class Properties
{
    public static Func<Guid, int> Combine(
        Func<Guid, int> f,
        Func<Guid, int> g)
    {
        return x => f(x) + g(x);
    }

    [Property]
    public void CombineIsAssociative(
        Func<Guid, int> f,
        Func<Guid, int> g,
        Func<Guid, int> h,
        Guid guid)
    {
        Assert.Equal(
            Combine(Combine(f, g), h)(guid),
            Combine(f, Combine(g, h))(guid));
    }
}

Output:

System.NotSupportedException:
  Unable to auto-generate System.IntPtr. You can use
  'GenX.defaults |> AutoGenConfig.addGenerator myGen |> GenX.autoWith'
  to generate types not inherently supported by GenX.auto.

This example fails when using with Hedgehog 0.12.x but runs fine with Hedgehog 0.11.x

[<Property>]
let ``should be ok``(value : int) =
    value = value

Here is the exception:

System.MissingMethodException
Method not found: 'Int32 Hedgehog.FailureData.get_Size()'.
   at InternalLogic.tryRaise(Report report)
   at XunitOverrides.PropertyTestInvoker.CallTestMethod(Object testClassInstance) in /home/runner/work/fsharp-hedgehog-xunit/fsharp-hedgehog-xunit/src/Hedgehog.Xunit/XunitOverrides.fs:line 11
   at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<<InvokeTestMethodAsync>b__1>d.MoveNext() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 257

I checked with using hedgehog-experimental and GenX.auto directly and it works, so the bug must be somewhere in this library.

The following dependencies are used:

"Hedgehog" Version="0.12.1"
"Hedgehog.Xunit" Version="0.3.0"
"xunit" Version="2.4.1"

This executes the test once:

    [Fact]
    public void ShouldMap_UserCompetition_Equivalent2()
    {
        var prop =
            from source in ForAll(GenX.autoWith<CompetitionDetailsCache>(CompetitionsAutoGenConfig.Config()))
            select source.ToUserCompetition(Mapper)
                         .Should()
                         .BeEquivalentTo(
                             source.ToUserCompetition());

        prop.ReCheck("50_17003736010433171956_2975322824293632487_000000000000000000101010111110111111011111110111111110111111111101111111111101111111111110111111111111101111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110");
    }

And this executes the test 128667 times, always with the same values:

    [Property, Recheck("50_17003736010433171956_2975322824293632487_000000000000000000101010111110111111011111110111111110111111111101111111111101111111111110111111111111101111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110111111111111111110")]
    public void ShouldMap_UserCompetition_Equivalent(CompetitionDetailsCache source)
    {
        source.ToUserCompetition(Mapper)
              .Should()
              .BeEquivalentTo(source.ToUserCompetition());
    }

Changing a number of tests in Property attribute (I tried [Property(1)]) doesn't seem to have any affect on this behaviour.

Expected: test case is executed once in both cases :)

On nuget.org, the dependencies look correct:

But in Visual Studio, it says there are no dependencies.

Why is that?

Currently we can supply AutoGenConfig on two levels:

1. Using Properties attribute on a class/module
2. Using Property attribute on a specific test

Current behaviour is that the config that is specified by Property attribute "wins".

I suggest that we add the ability to "merge" configs so that it'd be possible to supply a bunch of generators with Properties attribute, but the AutoGenConfig that is supplied for an individual test would be "appended" to the upper-level config.

This would probably mean that we make AutoGenConfig a semigroup, which has its own advantages, too.

I find that in most of my tests, the set of test parameters have invariants that must be satisfied. For example, the property Foo of the object in the second argument must be larger than the value in first argument. Not to mention that in many tests, values must satisfy some criteria – for example, a value must be positive in order for the test to make sense.

I am used to the property { ... } computation expression, where I generate one value at a time and can specify the exact requirements for each value. Thess requirements may be based on previously generated values, in order to preserve invariants in the whole set of values.

Based on the readme in this repo, I understand how to accomplish the same with Hedgehog.Xunit using AutoGenConfig. However, if I am to define a separate generator config for more or less every test function, then I don't really see the benefit over simply using property { ... } |> Property.check and generating all values inside there.

Furthermore, to get the same flexibility with Hedgehog.Xunit (specifically, if multiple arguments have the same type but different requirements, such as one positive and one negative integer), I would need to wrap the arguments in a custom type (e.g. a record) to be able to enforce the necessary invariants.

This leads me to believe that Hedgehog.Xunit is mostly useful if one only rarely has test-specific invariants or other requirements for test parameters.

Does that make sense (and do you agree), or have I missed something?

/cc @TysonMN

The project's current state looks awesome, and from what I can tell from its README, feature complete. Moving it into the core org would make it more discoverable 🚀

Does this package need to rely on xunit, and not just xunit.core (or xunit.abstractions, which it currently relies on)?

Asking because I only reference xunit.core v2.4.2 in my test project (it's all I need), but I was then unable to install Hedgehog.Xunit because it tried to install xunit 2.4.1, causing a version conflict. (I was able to work around it by installing xunit 2.4.2 explicitly, but unless Hedgehog.Xunit needs that specific package, it seems unnecessary.)