factor10 / intent Goto Github PK

Example: Suite class named "ToEqualTest" and the first context is "ToEqual".

We could have a convention where we take the class name and remove the "Test" suffix and use what remains as suite name. Such a convention should be possible to customize.

Allow toEqual for Map

Immutable + mutable, preferably comparable with each other.

I.e. exception in a via method. See #8 wrt TestFailure/TestError.

When implementing Intent[Unit] you should not have to implement def emptyState or take an argument in the test block "test" in { ignored => .. }.

Every now and then (fairly often actually) the following test fails:

[info] [ERROR] intent.runner.TestSuiteRunnerTest >> TestSuiteRunner >> running the OneOfEachResultTestSuite >> with a registered event subscriber >> should publish 3 events (1 ms)
[info] 	Expected size to be 3 but was 2

I believe the reason is that sometimes when the test execution Future[TestSuiteResult] is fulfilled, the test Await.result [1] happens before the test suite's .onComplete [2] that publish the event (which in turn increment the received event array).

[1] 46bf081#diff-f3e738aed58795fff682ea06952eed7fR54

[2] 46bf081#diff-b597a920904a06ca80b57af58adfe6c1R66

Not sure which syntax we want, but the need is there.

Drain the existing SBT runner by creating a standalone runner that can be reused for other build systems etc.

Go through the existing matchers and make sure errors are expressed clearly enough.

I believe the current DSL can be improved. E.g.:

class StatefulTest extends TestSuite with State:
  setup:
  - "an empty cart" Cart() as "myCart":
  - "another empty cart" Cart() as "anotherCart":
  where:
    - tabular:
      +
      * | "beachChair" | "sunScreen" | "total":
      -
      "quantity" |  2  |  3  |  5 :
      "price" |  2  |  3  |  5 :
  then:
  - sequentially:
    - "add two items" using "myCart" '{_.add(CartItem("Beach chair", data["beachChair"]["quantity"]))}:
    - parallelly:
       - ....
       - ....
       - sequentially:
         - ....
         - ....
    - "add another three items" '{state["myCart"].add(CartItem("Sunscreen", data["sunScreen"]["quantity"]))}:
  expect:
  - "contains 5 items" '{state["myCart"].totalQuantity == data["total"]["quantity"]} and:
  - "costing" using "myCart" '{_.totalPrice == data["total"]["price"]}:
  lastly:
  - "do some cleanup" using (_.close())

For table tests can they be specified like:

class MaxTest extends TestSuite with Stateless:
  expect:
    - "Max of a and b" '{ Math.max(data["a"], data["b"]) == data["c"] }

  where:
    - tabular:
      +
      | "a" | "b" | "c":
      -
      |  1  |  3  |  3 :
      |  7  |  4  |  7 :
      |  0  |  0  |  0 :

Support to match any instances are required.

The following test fails, since there is not equality check for type Data:

"for case classes":
      "can compare" in:
        class Data(value: Int)
        expect(Data(1)).toEqual(Data(1))

[error] /Users/markus/Development/factor10/dotty-intent/src/test/scala/intent/matchers/ToEqualTest.scala:117:41: no implicit argument of type intent.core.Eq[Data] was found for parameter eqq of method toEqual in trait ExpectGivens.
[error] I found:
[error]
[error]     this.tryEq[TInner, T]
[error]
[error] But method tryEq in trait EqGivens does not match type intent.core.Eq[Data].
[error]         expect(Data(1)).toEqual(Data(1))
[error]                                         ^
[error] one error found
[error] (Test / compileIncremental) Compilation failed
[error] Total time: 5 s, completed 19 feb. 2020 23:24:51

Preferably any types should be comparable with toEqual at least. Result should be a pretty-printed type and structural difference between expected and actual.

E.g. wrong number of tests, wrong number of failures.

Some times tests not only need to produce the correct results the results should be given within an acceptable time. So can performance testing be added which can:

• ensure core runs within a time-bound
• ensures code runs as fast as or faster than a benchmark price of code
  • sure code runs as fast or faster than the previous build
    • easily persist the previous results and load the data

For example:

expect(<...>).toThrow[IllegalArgumentException]()

With message (regex?)

expect(<...>).toThrow[IllegalArgumentException]("it failed")

Or with pattern matching:

expect(<...>).toThrow:
  case ex: IllegalArgumentException => expect(ex.getMessage).toMatch(...)

But there are two expects there... We could do it like whenComplete:

expectFailure(<...>):
  ex => ... // could do pattern matching here also

Or something like intercept in ScalaTest.

@eliasson thoughts?

Currently I only find a version for dotty 0.18 on maven central (https://search.maven.org/artifact/com.factor10/intent_0.18/0.1.0/jar). Can you also publish the 0.19 version? Or on which repository can I find it?

Things todo:

• Remove dotty part from name (ie. only name sbt-project and repository intent)
• Rewrite README.md with getting started.
•  Add docs/ folder (using dotty-docs tool) where design philosophy and usage can be expressed.

Build and publish of releases to sonatype should be automated.

Things to consider:

• Semantic versioning?
• Use github releases?
• Have a separate GH Action to verify consumption of package?

Should be avoided, but may be useful.

  expect.multiple(Seq(
    expect(...).toEqual(...),
    expect(...).toMatch(...)
  ))

NUnit's Assert.Multiple is not really equivalent since we need our tests to return an Expectation. So we need a compound Expectation here.

Perhaps a params array is nicer:

  expect.multiple(
    expect(...).toEqual(...),
    expect(...).toMatch(...)
  )

@eliasson thoughts?

It should be possible to focus or ignore hierarchies of tests.

E.g.:

"test" in:
  val f = Future { throw new Exception("oops") }
  whenComplete(f):
    r => ...

I haven't tested, but I have a feeling the exception isn't handled correctly.
A failed Future ought to be handled as a test failure (not a test error).

Make it possible to write fail(X) instead of using some sort of toEqual-hack.

Add documentation on the following subjects:

• Introduction, rational and example
• Matchers
• Table-driven tests
• Futures / async testing
• Async state

Currently the path up to the test is printed, not the final test name (in case of nested tests at least)

I.e. current invalid behaviour prints:

[info] [PASSED] intent.core.ObservableTestSuite >> Observable >> a misbehaving subscriber (0 ms)
[info] [PASSED] intent.core.ObservableTestSuite >> Observable >> a misbehaving subscriber (0 ms)
[info] [PASSED] intent.core.ObservableTestSuite >> Observable >> a misbehaving subscriber (1 ms)

This failed in the Dotty community build (currently based on 989f238)

[info] [FAILED] intent.util.DelayedFutureTest >> A DelayedFuture >> should be cancellable (383 ms)
[info] 	Expected false but found true (/tmp/3/community-build/community-projects/intent/src/test/scala/intent/util/DelayedFutureTest.scala:47:32)
[info] [FAILED] intent.util.DelayedFutureTest >> A DelayedFuture >> should have a default result after being cancelled (382 ms)
[info] 	Expected 0 but found 42 (/tmp/3/community-build/community-projects/intent/src/test/scala/intent/util/DelayedFutureTest.scala:55:35)

If a test match on one of Either[L, R] and the other occurs should intent handle that and provide a nice assertion error instead?

Same for all unmatched errors?

Applies primarily to whenComplete I think.

But it could also be relevant for normalt tests, doing stupid things like getting into infinite loops.

The timeout should be configuration. Mixing in a "patience" trait like in ScalaTest in one approach, just using a given is another.

class ToCompleteWithTest extends TestSuite with Stateless:
  "toCompleteWith" :
    "for successful Future" :
        "can be negated" in expect(Future.successful("foo")).not.toCompleteWith("foo")

Is expected to pass but fails with

[info] [ERROR] intent.matchers.ToCompleteWithTest >> toCompleteWith >> for successful Future >> can be negated (6 ms)
[info] 	Expected Future not to be completed with "foo"

See #7

Upon failure we create a TestFailure, but it only has a message. We could have Option[Throwable] there as well, but it needs to be handled in the runner.

Maybe merge TestFailure and TestError? Is there a reason to separate them?

Remove old tests and code (Main.scala) from the initial experiments.