Where is ListStack coming from? It does not seem to be jdk or kotlin-stdlib.

Imho landing page examples should self-contained and easy to reproduce.

Also even if the readme mentions the init-block and that everything needs to go inside, it would be easier to get started if all examples would be self-contained (full classes).

Example:

class MyException() : Throwable()

class Playground : FlatSpec() { init {
    "it" should "do something" {
        throw MyException()
    }

    "it" should "do something else" {
    }
}
}

In Intellij/gradle this gives an incorrect picture like if the first test suceeded correctly.

If a class MyRunException() : RuntimeException() instance is thrown instead, the test is tagged as failed as expected. Why don't Throwables cause tests to fail but require RuntimeException as base class?

I thought it may be a junit feature/problem after all but it does not seem to be the case:

public class Dummy {

    @Test
    public void runTest() throws Throwable {
        throw new Throwable();
    }

    @Test
    public void runAnotherTest() throws Throwable {
    }
}

Here runTest fails as expected:

Add some kind of describe/scenario spec
Add some kind of given/when/then (although when is a keyword)

At the moment config is only available per test case. But it would be useful to have a configuration per specification class, that could be overridden per test case if necessary. My suggestion is as follows:

Instantiating a TestConfig object directly and assign it to an overridden property :

class MySpec : StringSpec() {
    override val config = TestConfig(threads = 10, invocations = 100)
    ...
}

or using a factory method instead of a direct instantiation:

class MySpec : StringSpec() {
    override val config = config(threads = 10, invocations = 100)
    ...
}

(If TestConfig would be renamed to just Config both variants were almost the same from a users perspective.)

Another option would be to call a config method inside the init block and don't use an overridden property, but I prefer a property.

class MySpec : StringSpec() {
    init {
        config(threads = 10, invocations = 100)

        ...
    }
}

I think a class wide configuration should reside on the class level as in the examples with the property.

If I run all tests of the KotlinTest project, the execution (in IntelliJ 2016.1.3) stops with 109 of 110 tests. It appears the message:

[WARN] When comparing doubles consider using tolerance, eg: a shouldBe b plusOrMinus c

breaking change, needs to got to 2.0.0

The test:

class Factorial_KotlinTest_PropertyBased : PropertyTesting() {
  init {

    val algorithms = listOf(
        "iterative" to {x:Int -> iterative(x)},
        "reductive" to {x:Int -> reductive(x)},
        "naïve_recursive" to {x:Int -> naïve_recursive(x)},
        "tail_recursive" to {x:Int -> tail_recursive(x)}
    )

    forAll(algorithms) { a ->
      val f = a.second
      property(a.first + ":recurrence relation is true for all non-negative integer values").
          forAll{i: Int ->
            if (0< i && i < 500) { f(i) == (i.bigint * f(i - 1)) }
            else { true }
          }
    }

  }
}

was working fine 10 days ago. With the 1.3.1 release I thought I'd update. I thought:

class Factorial_KotlinTest_PropertyBased : StringSpec() {
  init {

    "Factorial recurrence relation works for all algorithms for all non-negative integers" {

      val algorithms = listOf(
          "iterative" to { x: Int -> iterative(x) },
          "reductive" to { x: Int -> reductive(x) },
          "naïve_recursive" to { x: Int -> naïve_recursive(x) },
          "tail_recursive" to { x: Int -> tail_recursive(x) }
      )

      forAll(algorithms){a ->
        val f = a.second
        forAll{i: Int ->
          if (0 < i && i < 500) { f(i) == (i.bigint * f(i +1)) }
          else { true }
        }
      }
    }

  }
}

to be the replacement. However it is clearly broken since it passes and clearly should not. There is an error introduced which means it should fail.

The lately added features like config (#14) should be documented. Without documentation they are not visible to most users and hence not useful.

forAll(x) {…}

works fine for:

val x = listOf(…)

but fails for:

val x = arrayOf(…)

there seems no reason for this.

The scala-way of testing collections does not yet work correctly in kotlintest:

listOf("foo", "bar_1", "bar_b2") should contain "sdf" 
//nor
listOf("foo", "bar_1", "bar_b2") should be listOf("lala")

See http://www.scalatest.org/user_guide/using_matchers#javaCollectionsAndMaps

Some interfaces exist only to hold concrete methods. In languages like Java and Scala this is necessary, but Kotlin supports top level functions, so this code can be slightly reduced by removing the interfaces.

The following interfaces could be removed: Inspectors, Eventually, CollectionMatchers, ExceptionMatchers, IntMatchers, LongMatchers, MapMatchers, StringMatchers, TypeMatchers. This would be a breaking change.

Consider adding a junit style "function per test" spec for people who don't want to nest things in init blocks.

class MethodSpecTest : MethodSpec() {
  fun testone() {
  }
  fun someOtherTest() { 
  }
}

It would be nice to have a way to do the opposite of the "ignore" option in the test config so that only one test is run and all others are temporarily disabled.

Is there a nightly build, or are people just compiling from master/HEAD locally? If the former is there a repository that can be used in Gradle builds? If the latter is there a standard policy of where to put the artefact for use with Gradle?

It would be useful to be able to mark test as ignored. Something like @Ignore from JUnit, but not necessarily an annotation.

Sometimes you want to execute only certain tests. Something like ScalaTest's Tagging or TestNG's Groups, preferably with exclusion would be handy.

Was thinking of adding a new spec, called StringSpec which just requires a single string, eg,

"haveKey should test that a map has a key" {
        val map = mapOf(Pair(1, "a"), Pair(2, "b"))
        map should haveValue("a")
}

Wdyt @helmbold

Each release should have a simple wiki page with release notes. This is not only useful for existing users but a good "marketing" instrument, too. The Rust team gets a lot of attention every six weeks, because they write down what's new.

im just trying it out in a gradle project.
in intellij everything is fine, but when running gradle test it does not work.
eg this test passes in gradle but fails in IJ

class AnotherTest() : FunSpec(), Logging {
override val log: Logger by lazyLogger()

init {
    test("fails") {
        log.debug("log sth")
        false shouldBe true
    }
}

override fun afterAll() {
    super.afterAll()
}
}

test result output is

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="AnotherTest" tests="0" skipped="0" failures="0" errors="0" timestamp="1970-01-01T00:00:00" hostname="ytu06" time="0.0">
<properties/>
<system-out><![CDATA[17:09:47.987 [Test worker] DEBUG kotlin.AnotherTest - log sth
]]></system-out>
<system-err><![CDATA[]]></system-err>
</testsuite>

I've just started using this, so apologies if I'm missing something.

When I write tests that pass, all is good and happy. However, if the tests fails then I get two runs of the test reported, one that passed and one that failed with the actual error. This doesn't actually cause problems - the overall test still failed - but it looks weird.

For example, the test:

class DemoTest : WordSpec() {
    init {
        "calculator" should {
            "perform addition" {
                (1 + 1).shouldEqual(3)
            }
        }
    }
}

Gives the following output in Maven:

Tests in error:
calculator.should perform addition(calculator)
  Run 1: PASS
  Run 2: 2 did not equal 3

Fixing the test so that it passes appears to only run the test once, and that is what I'd expect here too.

• invocation count - the number of different values that should be fed in
• minsuccessful - the mininum number of attempts that must pass
• threads - number of threads to use in the executor pool

At the moment it is only possible to test whether an exception of a certain was thrown:

shouldThrow<IllegalAccessException> {
  // code in here that you expect to throw a IllegalAccessException
}

But sometimes it is necessary to test some details of the exception like the message. This could be done by provider an overloaded shouldThrow method, which would provide a reference to the exception.

shouldThrow<IllegalAccessException> { exception ->
  // code in here that you expect to throw a IllegalAccessException
  exception.message should start with "Something went wrong"
}

See #35

The property-based testing seems to be able to automatically generate integers (and I assume floats, etc), but I need to be able to generate sets of strings. I am guessing this is handled as yet as I am getting lots of incomprehensible messages when I try it.

Is there an idiomatic way of handling this immediately?

shouldThrow<StackOverflowError> { … }

and

expecting(StackOverflowError::class){ … }

appear not to catch StackOverflowError it propagates out to the runner.

cf. the KotlinTest test in https://github.com/russel/Factorial/tree/master/Kotlin

I've pretty good experience writing unit test methods starting with should. This simple words leads to much better names compared to test. Therefore I suggest, to support another style like fun spec, but using the word should. Example:

should("remove the last element from stack") {
  val stack = ListStack<String>()
  stack.push("hello")
  stack.push("world")
  stack.size() shouldBe 2

  val element = stack.pop() 

  element shouldBe "world"
  stack.size() shouldBe 1
}

Compared to the existing style with test it is even more concise, since it is already known what object is tested in a certain unit test class.

should("remove the last element from stack")
test("ListStack.pop should remove the last element from stack")

Ie, to assert that a collection is a single element of x

Kotlin is at 1.0.3 by now, while KotlinTest is still using 1.0.1-2, which causes intelliJ to fail the gradle sync because of mismatching dependency version. I don't really want to go back to an earlier Kotlin version for my project and miss out on the faster compiling just to use KotlinTest...

It would be easier to contribute, if each release (1.2, 1.3, 2.0 ...) would have an own branch. I have an idea which would break backwards compatibility and that must be part of a new major release. At the moment I wouldn't be able to make a pull request for it, because it would immediately be part of the next minor version.

ScalaTest works best when used with ScalaCheck, this way you get example-based and property-based testing in the same test framework. If KotlinTest is ScalaTest for Kotlin (which is seems to be :-) ) is there a Kotlin equivalent of ScalaCheck (let's call it KotlinCheck), and is it integrated into KotlinTest as ScalaCheck is in ScalaTest?

Specs2 in scala runs its tests using akka. We don't have to do that (could use an executor pool) but atm we're not taking advantage of multi cores to run the tests concurrently. @helmbold

Similar to the trait provided by scalatest

Writing KotlinTest tests in IntelliJ editor works fine. Executing individual KotlinTest files in IntelliJ runs the tests. However trying to run "All Tests" in the test module of an IntelliJ IDEA project with KotlinTest tests results in no tests being found.

QuickCheck and Hypothesis, and all good variants of these, support shrinking: whenever an error occurs in a property test, a chop search is undertaken to find the smallest value from the set that exhibits the failure. Does KotlinTest do this? If not can it be added to the agenda for fairly quick implementation. Property-based testing frameworks are evaluated on supporting domain sampling based on static type, ability to specify bizarre generators, and shrinking.

Add scalatest style Inspectors, to allow:

forAll
forAny
forAtLeast(x)
forAtMost(x)
forOne
forNone
forExactly(x)
forBetween(x,y)

In class TestBase is oneInstancePerTest set to false by default. Although a testcase should be stateless and independent from other tests, it is a safer default to create a new instance per test case.

This is, by the way, the default in some other frameworks, too. In JAX-RS for example is a resource object created per request, although it would be possible to have stateless resources and reuse a resource object for many requests. I guess they have chosen this default, because it helps to avoid bugs.

Some form of parametrized tests like ScalaTests table driven test would be nice.

What should be the FreeSpec example in the README.md is incorrectly titled as FlatSpec, and the description as well as the code sample both show FlatSpec as a name.

As a newcomer to this library this gave me a hell of a time figuring out why the IDE would flag the minus sign even when I just copied and pasted the example code, until I finally noticed that FlatSpec had two conflicting sample sections and FreeSpec had none.

It is still not possible to compile KotlinTest under Windows due to name clashes (be vs. Be). See this issue.

Whenever you compare two values like this: d shouldBe 0.0 you get this warning:

[WARN] When comparing doubles consider using tolerance, eg: a shouldBe b plusOrMinus c

I have a project with a lot of testcases that do indeed need exact double matches with no tolerance, and this output spamming the console for every double comparison is kind of annoying. I would appreciate if there was a way to disable that output - a possible way that goes in line with the testing syntax might be to use a syntax like this: d shouldBe exactly(0.0)