com-lihaoyi / utest Goto Github PK

View Code? Open in Web Editor NEW

481.0 481.0 80.0 1.79 MB

A simple testing framework for Scala

Scala 98.97% Shell 0.94% Java 0.09%

utest's People

Contributors

Stargazers

Watchers

utest's Issues

lazy vals crash compiler

Yo. It seems lazy vals within tests crash scalac.
It happens when compiling for JVM and JS.

Reproduction:

object What { def mf1 = 2 }
object SomeTest extends TestSuite {
  override def tests = TestSuite {
    import What._
    lazy val subj = mf1
  }
}

You can see big error messages if you want in scala-js/scala-js#1580

Arg parsing

object DefaultFormatter{
  def apply(implicit args: Array[String]) = {
    val color = utest.util.ArgParse.find("--color", _.toBoolean, true, true)
    val truncate = utest.util.ArgParse.find("--truncate", _.toInt, 50, 50)
    val trace = utest.util.ArgParse.find("--trace", _.toBoolean, false, true)

    new DefaultFormatter(color, truncate, trace
    )
  }
}
/**
 * Default implementation of [[Formatter]], also used by the default SBT test
 * framework. Allows some degree of customization of the formatted test results.
 */
class DefaultFormatter(color: Boolean = true, 
                       truncate: Int = 30,
                       trace: Boolean = false) extends Formatter{

Array[String] is way too general a type to be passing around as an implicit.
Why are default values specified - with different defaults - in two or more places?
Is there some way to pass these args other than the command line? I didn't have any luck passing them as test options from my build.sbt. I can't see any of my errors because they're cut off at one line. Here's a typical failed test run:

[info]     valdef       Failure(java.util.regex.PatternSyntaxExceptio...
[info] Tests: 2
[info] Passed: 1
[info] Failed: 1

I love reinventing stuff as much as the next guy but command line option parsing should be reinvented well or not reinvented.

scala-js should remains arms-length from utest

I understand the temptation to avoid project proliferation and the buzzkill it is when users start showing up with their fickle demands. However, I had to strip all the javascript code out of utest in order to continue. My travis builds were failing because they couldn't find the scala-js snapshot plugin. After floundering around in sbt for the appropriate amount of time trying to figure out where was that snapshot dependency, I decided it made no sense to shave this yak given that I have no interest in scala-js whatsoever, I'm here for the test framework.

I'm going to guess you need no convincing that utest shouldn't require additional snapshot dependencies which involve an unrelated project not even in the same language.

For now I put 'git rm' to work and switched my source dependency to https://github.com/paulp/utest/tree/2.11 after which travis went green.

Unable to pass jvm parameters to sbt test task

Hi,

first off, utest is really awesome and pragmatic, as the rest of your all your OSS :-)

As I'm new to sbt I have hard times to do something like this :

javaOptions in test += "-Xmx=1024G -Dlog4j.configuration=log4j-server.properties"

The JVM parameters are not propagated to the test process, otherwise it would blow up due to huge Xmx value.

Any idea how to deal with that ?

Compiling utest with scala-js 0.6.0-SNAPSHOT and scala 2.11.0, need help

Hey, it's 3rd time I'm trying to do that, everytime I do a PR to scala-js SNAPSHOT and I need to use it I fail to recompile utest with it, always from some weird reasons.

For instance now the compiler cannot see scala.scalajs.js.Any#toArray implicit definition and I get stuff like :

utest/js/PlatformShims.scala:52: type mismatch;
[error]  found   : scala.scalajs.js.Array[String]
[error]  required: scala.Array[String]
[error]       args,
[error]       ^

Would you please help me with it? I kinda need to be on cutting edge when working with scala-js... But it takes a lot of time unfortunately

--color=false ignored in Js Runner

you close #50 but I found a bug in your workaround 😉

As the title says, passing --color=false to the tests, when run on Rhino, is ignored.

To see, run sbt test on my project https://github.com/fommil/kerbal-calculator and see that no special characters are printed in the JVM project, but special characters are printed in the Rhino run.

This branch definitely exhibits it: https://github.com/fommil/kerbal-calculator/tree/a431c0a7e110faa7bb864c290de1cc7503ad8a4a

Colours bleeding to new line when truncating in DefaultFormatter

Because you truncate after you've appended Console.RESET, colours bleed over to the next line, which is very confusing

See: https://github.com/lihaoyi/utest/blob/e21a15a963d34b8117bdcfbc06caa6db108609f2/shared/main/scala/utest/Formatter.scala#L49

Better `compileError` helper

compileError is cool, but it leaves open what (basically all my libraries) I really want: to check that the error is in the right place, and that certain things are in the error message.

A higher-level compileErrorContains macro/method would eliminate this duplication throughout my code.

Example of property-based testing with uTest

It would be nice to have an example of doing property-based testing in uTest, either by using ScalaCheck, or some other means.

Exception when running testsuite from Intellij IDEA

When i run the following testsuite from within Intellij:

package se.oobo.validation

import utest._
import utest.ExecutionContext.RunNow

object ValidationTest extends TestSuite {
  override def tests = TestSuite {
    'sometest - {
      val x = 3
      val y = 2
      assert(x != y)
    }
  }
}

...i get this exception:

Exception in thread "main" java.lang.IllegalStateException: Test that returns Future must be run asynchronously in Scala.js, see TestTreeSeq::runAsync
at utest.PlatformShims$.await(PlatformShims.scala:18)
at utest.framework.TestTreeSeq.run(Model.scala:141)
at org.jetbrains.plugins.scala.testingSupport.uTest.UTestRunner.runTestSuites(UTestRunner.java:198)
at org.jetbrains.plugins.scala.testingSupport.uTest.UTestRunner.main(UTestRunner.java:258)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)

Process finished with exit code 1

Any ideas?

Can't compile - "symbol value x$54 does not exist" ...

I've more or less minimized a code that fails to compile. If I e.g. remove the doAction wrapping, or do other changes, it compiles fine.

object FooTest extends utest.TestSuite {
  case class X(dummy: Int = 0, x: Int = 0)

  protected[this] def doAction(action: ⇒ Any): Unit = {
    action
  }

  val tests = TestSuite {
    * - {
      doAction {
        X(x = 1)
      }
    }
  }
}

Compiler failure:

Error:scalac: 
  symbol value x$54 does not exist in com.measprog.das.performance.FooTest$$anonfun$23$$anonfun$apply$16.apply
     while compiling: /home/td/projects/mmd-gui/src/test/scala/com/measprog/das/performance/ProtobufPerformanceTest.scala
        during phase: icode
     library version: version 2.11.4
    compiler version: version 2.11.4
  reconstructed args: -nobootcp -javabootclasspath : -deprecation -feature -classpath /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/javazic.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rhino.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/compilefontconfig.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/management-agent.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/java-atk-wrapper.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/pulse-java.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/localedata.jar:/home/td/projects/mmd-gui/target/scala-2.11/test-classes:/home/td/projects/mmd-gui/target/scala-2.11/classes:/home/td/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.1.2.jar:/home/td/.ivy2/cache/ch.qos.logback/logback-core/jars/logback-core-1.1.2.jar:/home/td/.ivy2/cache/com.google.protobuf/protobuf-java/bundles/protobuf-java-2.5.0.jar:/home/td/.ivy2/cache/com.lihaoyi/utest-runner_2.11/jars/utest-runner_2.11-0.2.4.jar:/home/td/.ivy2/cache/com.lihaoyi/utest_2.11/jars/utest_2.11-0.2.4.jar:/home/td/.ivy2/cache/com.miglayout/miglayout-core/jars/miglayout-core-4.2.jar:/home/td/.ivy2/cache/com.miglayout/miglayout-swing/jars/miglayout-swing-4.2.jar:/home/td/.ivy2/cache/com.typesafe.scala-logging/scala-logging_2.11/jars/scala-logging_2.11-3.1.0.jar:/home/td/.ivy2/cache/net.sandrogrzicic/scalabuff-runtime_2.11/jars/scalabuff-runtime_2.11-1.3.8.jar:/home/td/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.11.4.jar:/home/td/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.11.2.jar:/home/td/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.11.4.jar:/home/td/.ivy2/cache/org.scala-lang.modules/scala-parser-combinators_2.11/bundles/scala-parser-combinators_2.11-1.0.2.jar:/home/td/.ivy2/cache/org.scala-lang.modules/scala-xml_2.11/bundles/scala-xml_2.11-1.0.2.jar:/home/td/.ivy2/cache/org.scala-sbt/test-interface/jars/test-interface-1.0.jar:/home/td/.ivy2/cache/org.scalacheck/scalacheck_2.11/jars/scalacheck_2.11-1.11.6.jar:/home/td/.ivy2/cache/org.scalaz/scalaz-concurrent_2.11/bundles/scalaz-concurrent_2.11-7.1.0.jar:/home/td/.ivy2/cache/org.scalaz/scalaz-core_2.11/bundles/scalaz-core_2.11-7.1.0.jar:/home/td/.ivy2/cache/org.scalaz/scalaz-effect_2.11/bundles/scalaz-effect_2.11-7.1.0.jar:/home/td/.ivy2/cache/org.slf4j/slf4j-api/jars/slf4j-api-1.7.7.jar:/home/td/.ivy2/cache/org.specs2/specs2-common_2.11/jars/specs2-common_2.11-2.4.11.jar:/home/td/.ivy2/cache/org.specs2/specs2-core_2.11/jars/specs2-core_2.11-2.4.11.jar:/home/td/.ivy2/cache/org.specs2/specs2-matcher_2.11/jars/specs2-matcher_2.11-2.4.11.jar:/home/td/.ivy2/cache/io.reactivex/rxjava/jars/rxjava-1.0.2.jar:/home/td/.ivy2/cache/io.reactivex/rxscala_2.11/jars/rxscala_2.11-0.23.0.jar:/home/td/.ivy2/local/info.td/fast-scala-plots-old_2.11/0.7.2-SNAPSHOT/jars/fast-scala-plots-old_2.11.jar:/home/td/.ivy2/local/info.td/td-scala-common_2.11/0.13.2-SNAPSHOT/jars/td-scala-common_2.11.jar -unchecked
  last tree to typer: Ident(x$55$1)
       tree position: line 137 of /home/td/projects/mmd-gui/src/test/scala/com/measprog/das/performance/ProtobufPerformanceTest.scala
            tree tpe: Int
              symbol: value x$55$1
   symbol definition: x$55$1: Int (a TermSymbol)
      symbol package: com.measprog.das.performance
       symbol owners: value x$55$1 -> constructor FooTest$$anonfun$23$$anonfun$apply$16$$anonfun$24 -> <$anon: Function0>
           call site: <$anon: Function0> in package performance in package performance
== Source file context for tree position ==
   134     * - {
   135       doAction {
   136         X(x = 1)
   137       }
   138     }
   139   }
   140 }

intercept{} and assertMatch{} should have the cool/useful error messages too

Currently only assert/eventually/continually have them, but that's more laziness than any real design decision. There's no reason these guys shouldn't have it too.

Republish for scalajs-0.6.1

To ensure that uTest users benefit from scala-js/scala-js#1506 (and avoid symptoms seen in scala-js/scala-js#1514), uTest needs to be republished for scalajs-0.6.1.

Stack traces aren't being cutoff

Really, really long stack traces.

I've tried this with

μTest 0.3.0
Scala.JS 0.6.0 and 0.6.1
PhantomJS 1.9.8 and 2.0.0

With Phantom.JS 2.0.0 it's like this:

[info]        java.lang.AssertionError: assertion failed
[info]  at createException__p1__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:60630:25)(Unknown Source)
[info]  at fillInStackTrace__jl_Throwable (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:85746:78)(Unknown Source)
[info]  at init___T__jl_Throwable (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:85843:38)(Unknown Source)
[info]  at init___T (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:97597:59)(Unknown Source)
[info]  at init___O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:105504:54)(Unknown Source)
[info]  at assert__Z__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:99568:53)(Unknown Source)
[info]  at Lxxx_webapp_base_test_BaseTestUtil$class__assertEq__Lxxx_webapp_base_test_BaseTestUtil__s_Option__O__O__Lscalaz_Equal__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:33247:39)(Unknown Source)
[info]  at Lxxx_webapp_base_test_BaseTestUtil$class__assertEq__Lxxx_webapp_base_test_BaseTestUtil__T__O__O__Lscalaz_Equal__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:33213:141)(Unknown Source)
[info]  at apply__T2__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:157742:136)(Unknown Source)
[info]  at apply__O__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:157729:20)(Unknown Source)
[info]  at file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:86795:72(Unknown Source)
[info]  at apply__O__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:101839:23)(Unknown Source)
[info]  at sc_IndexedSeqOptimized$class__foreach__sc_IndexedSeqOptimized__F1__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:55520:18)(Unknown Source)
[info]  at foreach__F1__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:237158:81)(Unknown Source)
[info]  at foreach__F1__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:86793:32)(Unknown Source)
[info]  at xxx$webapp$client$app$ui$xxx$LogicTest$$anonfun$$anonfun$$anonfun$$test$1__Lxxx_webapp_base_data_xxx__Lxxx_webapp_client_app_ui_xxx_xxx$xxx__F1__sc_Seq__Lscalaz_Equal__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:157521:22)(Unknown Source)
[info]  at apply__T2 (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:157592:220)(Unknown Source)
[info]  at apply__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:157570:24)(Unknown Source)
[info]  at run__sci_List__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:51154:53)(Unknown Source)
[info]  at run__sci_List__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:51150:100)(Unknown Source)
[info]  at run__sci_List__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:51150:100)(Unknown Source)
[info]  at run__sci_List__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:51150:100)(Unknown Source)
[info]  at apply__s_Option__s_util_Try (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:174952:195)(Unknown Source)
[info]  at apply__O__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:174947:42)(Unknown Source)
[info]  at map__F1__s_util_Try (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:199301:65)(Unknown Source)
[info]  at file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:53438:41(Unknown Source)
[info]  at apply__O__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:101839:23)(Unknown Source)
[info]  at run__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:100336:34)(Unknown Source)
[info]  at execute__jl_Runnable__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:85056:20)(Unknown Source)
[info]  at executeWithValue__s_util_Try__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:100364:44)(Unknown Source)
[info]  at dispatchOrAddCallback__p2__s_concurrent_impl_CallbackRunnable__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:198841:49)(Unknown Source)
[info]  at onComplete__F1__s_concurrent_ExecutionContext__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:198972:72)(Unknown Source)
[info]  at s_concurrent_Future$class__map__s_concurrent_Future__F1__s_concurrent_ExecutionContext__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:53435:58)(Unknown Source)
[info]  at apply__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:174896:136)(Unknown Source)
[info]  at apply__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:174909:41)(Unknown Source)
[info]  at liftedTree1$1__p1__s_util_Try (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:86529:72)(Unknown Source)
[info]  at run__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:86524:50)(Unknown Source)
[info]  at execute__jl_Runnable__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:85056:20)(Unknown Source)
[info]  at apply__F0__s_concurrent_ExecutionContext__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:53598:35)(Unknown Source)
[info]  at runFuture__F2__sc_Seq__sc_Seq__s_concurrent_Future__s_concurrent_ExecutionContext__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:51273:115)(Unknown Source)
[info]  at apply__Lutest_util_Tree__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:175319:124)(Unknown Source)
[info]  at utest$framework$TestTreeSeq$$anonfun$$runChildren$1__sc_Seq__sci_List__I__s_concurrent_Future__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:174875:58)(Unknown Source)
[info]  at apply__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:174900:142)(Unknown Source)
[info]  at apply__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:174909:41)(Unknown Source)
[info]  at liftedTree1$1__p1__s_util_Try (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:86529:72)(Unknown Source)
[info]  at run__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:86524:50)(Unknown Source)
[info]  at execute__jl_Runnable__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:85056:20)(Unknown Source)
[info]  at apply__F0__s_concurrent_ExecutionContext__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:53598:35)(Unknown Source)
[info]  at runFuture__F2__sc_Seq__sc_Seq__s_concurrent_Future__s_concurrent_ExecutionContext__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:51273:115)(Unknown Source)
[info]  at apply__Lutest_util_Tree__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:175319:124)(Unknown Source)
[info]  at utest$framework$TestTreeSeq$$anonfun$$runChildren$1__sc_Seq__sci_List__I__s_concurrent_Future__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:174875:58)(Unknown Source)
[info]  at apply__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:174900:142)(Unknown Source)
[info]  at apply__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:174909:41)(Unknown Source)
[info]  at liftedTree1$1__p1__s_util_Try (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:86529:72)(Unknown Source)
[info]  at run__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:86524:50)(Unknown Source)
[info]  at execute__jl_Runnable__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:85056:20)(Unknown Source)
[info]  at apply__F0__s_concurrent_ExecutionContext__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:53598:35)(Unknown Source)
[info]  at runFuture__F2__sc_Seq__sc_Seq__s_concurrent_Future__s_concurrent_ExecutionContext__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:51273:115)(Unknown Source)
[info]  at runAsync__F2__sc_Seq__sc_Seq__s_concurrent_ExecutionContext__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:51268:119)(Unknown Source)
[info]  at runUTestTask__sc_Seq__sc_Seq__T__Lsbt_testing_EventHandler__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:85268:104)(Unknown Source)
[info]  at file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:85137:103(Unknown Source)
[info]  at apply__O__O__O__O__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:101959:23)(Unknown Source)
[info]  at apply__T__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:175604:67)(Unknown Source)
[info]  at apply__O__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:175580:44)(Unknown Source)
[info]  at file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:56346:59(Unknown Source)
[info]  at apply__O__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:101839:23)(Unknown Source)
[info]  at foreach__F1__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:229849:18)(Unknown Source)
[info]  at sc_TraversableLike$class__map__sc_TraversableLike__F1__scg_CanBuildFrom__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:56344:24)(Unknown Source)
[info]  at map__F1__scg_CanBuildFrom__O (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:229964:96)(Unknown Source)
[info]  at executeInternal__p1__Lsbt_testing_EventHandler__ALsbt_testing_Logger__s_concurrent_Future (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:85482:68)(Unknown Source)
[info]  at execute__Lsbt_testing_EventHandler__ALsbt_testing_Logger__F1__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:85406:110)(Unknown Source)
[info]  at execute__p2__sjs_js_Dynamic__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:65975:73)(Unknown Source)
[info]  at handleMsgImpl__T__F0__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:65740:42)(Unknown Source)
[info]  at handleMsg__p1__T__V (file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:9385:33)(Unknown Source)
[info]  at file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:9272:33(Unknown Source)
[info]  at file:///tmp/phantomjs-launcher-webpage655707619739411965.html:89:15(Unknown Source)

With Phantom.JS 1.9.8 it's the same except without the class/method info. So

[info]        java.lang.AssertionError: assertion failed
[info]     at file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:60630(Unknown Source)
[info]     at file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:85746(Unknown Source)
[info]     at file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:85843(Unknown Source)
[info]     at file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:97597(Unknown Source)
[info]     at file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:105504(Unknown Source)
[info]     at file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:99568(Unknown Source)
[info]     at file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:33247(Unknown Source)
[info]     at file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:33213(Unknown Source)
[info]     at file:///home/golly/xxx/projects/xxx/Code/webapp-client/target/scala-2.11/webapp-client-test-fastopt.js:157742(Unknown Source)
...

import utest._ dumps too much rubbish in your namespace

In particular, it brings in util, which keeps shadowing scala.util. There's a whole bunch of other identifiers that it dumps in your namespace by default that don't need to be there. All this stuff could easily be hidden out of sight.

Add support for Custom test configuration

Hi,

I'm trying to create multiple test configurations because async tests require PhantomJS(autoExit =false), but I think that utest doesn't support setting a filter in testOptions, at least I couldn't figure that out. I was playing with this setup :

  lazy val AsyncTest = config("async") extend(Test)

  lazy val js =
    project.in(file("js"))
      .configs(AsyncTest)
      .settings(scalaJSSettings:_*)
      .settings(utestJsSettings:_*)
      .settings(inConfig(AsyncTest)(Defaults.testSettings ++ utestJsSettings) :_*)
      .settings(
        persistLauncher in AsyncTest := false,
        testOptions in AsyncTest := Seq(Tests.Filter(s => s.contains("Async"))),
        postLinkJSEnv in AsyncTest := new PhantomJSEnv(autoExit = false),
       )

But the testOptions never works, either all tests are executed or none is.

Do you have any idea how to make it work, or what might be the problem ?

Changes for SJS 0.6.0?

Sorry if I missed it in the docs... but what has changed in the required config for using utest 0.3.0 with SJS 0.6.0? Obviously, there is no utest sbt plugin anymore (?); but after I've I added the utest library dependency and the new TestFramework(...) entry to my build.sbt, my test cases aren't executed anymore. BTW: I've upgraded from utest 0.2.5-RC1.

Failing tests don't trigger SBT Error

When a test fails sbt still outputs success, this makes it hard for automated build tools to detect that there was a failure.

Here is what utest does on failure:

...
[info]     test2        Failure(utest.AssertionError: (assert(a == b)...
[info] Tests: 4
[info] Passed: 3
[info] Failed: 1
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0
[success] Total time: 1 s, completed Apr 19, 2014 7:59:20 PM

Versus

ScalaTest

[info] TestSampleMacros:
[info] - NOPTest *** FAILED ***
[info]   org.scalatest.exceptions.TestFailedException was thrown. (TestSampleMacros.scala:14)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   ...
[info]   at cgta.macros.TestSampleMacros$$anonfun$1.apply$mcV$sp(TestSampleMacros.scala:14)
[info]   at cgta.macros.TestSampleMacros$$anonfun$1.apply(TestSampleMacros.scala:14)
[info]   at cgta.macros.TestSampleMacros$$anonfun$1.apply(TestSampleMacros.scala:14)
[info]   at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info]   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   ...
[info] ScalaTest
[info] Run completed in 111 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***
[error] Failed: Total 1, Failed 1, Errors 0, Passed 0
[error] Failed tests:
[error]     cgta.macros.TestSampleMacros
[error] (macros/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 1 s, completed Apr 19, 2014 8:04:09 PM

macros shouldn't rely on predef implicits

This is a godsend so far, thanks.

My next issue is that I mask a lot of the implicits which scala puts in the default scope. This fails quite mysteriously with utest, presumably because it is generating code which depends upon finding those implicits - in particular turning a String into a Seq[Char]. It would be super helpful if the macro trees didn't contain any hidden dependencies like that.

What I get without working around is:

[error] /r/psp-view/src/test/scala/ViewSpec.scala:23: type mismatch;
[error]  found   : Char('.')
[error]  required: String
[error]   val tests = TestSuite {
[error]                         ^
[error] one error found

I work around by un-shadowing Predef for that test, but at that point I'm no longer testing what I intend to test.

Port to scala 2.11

I'm kind of in the market for something like this but 2.10 is no longer an option for me. Just throwing it out there. Thanks.

please don't output special characters

(it may sound weird that I can report this bug in the light of #49, but utest can run my tests if I have a pure JVM project.)

I do all my editing and use of sbt via emacs (and ENSIME). The output of sbt is displayed in a buffer and special characters are not interpreted by default (the typical emacs philosophy is to enrich output by parsing it and adding faces/links/functions to special regions)

Hence, my tests look like this

[info] -----------------------------------Results-----------------------------------
[info] com.github.fommil.kerbal.SolverSpec      32mSuccess0m
[info]     should solve for a Kerbin to Mun 10 payload      32mSuccess0m
[info] com.github.fommil.kerbal.EngineSuite     32mSuccess0m
[info]     should return valid fuel tanks for the Poodle engine     32mSuccess0m
[info]     should return valid fuel tanks for the KR-1x2 engine     32mSuccess0m
[info] Failures:

where 32mSuccess0m is pretty ugly.

Tests using scalatags

I have a test in my scalajs project that calls code that creates an image element using scalatags and the DOM renderer.. When I try to run the test I get a big stacktrace with TypeError: Cannot call method "createElementNS" of undefined at the top. Sounds like there is no DOM? I'm not sure what is going on.

Thanks!

Port uTest to Scala.js 0.6 and its new test interface

I will take care of this as a means to test the interface. See scala-js/scala-js#1073.

not supported on Java 6

one can expect to see this sort of stack trace on Java 6 as you're relying on a constructor that was added in JDK6:

java.lang.NoSuchMethodError: java.lang.AssertionError: method <init>(Ljava/lang/String;Ljava/lang/Throwable;)V not found
    at utest.AssertionError.<init>(Errors.scala:25)
    at utest.asserts.package$.assertError(package.scala:19)

Allow ? in addition to - for tests that must be explicitly invoked

? could be added in addition to - on the string/symbol implicits to indicate that a test should be ignored unless specifically called, this is useful to when you are iterating on development, or when you want to show a particular feature.

Also if a test is failing for a known reason you want to defer until later you can flip it off by using this as well, it's really easy to just flip the - to a ?.

This is roughly equal to using @ignore explicit in other frameworks

(http://www.nunit.org/index.php?p=explicit&r=2.4.8)

Provide a way to run multiple tests at a time

This would be especially useful for numbered tests. e.g. something like

scalaParser/test-only -- scalaParser.SyntaxTest.unit.{6, 7, 8}
scalaParser/test-only -- scalaParser.SyntaxTest.unit.{6...8}

wrong "Results" summary for "fork:=true"

Hi! I just created an empty project, copy-pasted a demo Test, but I'm getting a wrong "Results" summary.

Code:

package net.pointsgame.debug

import utest._

object MyTest extends TestSuite {
    val tests = TestSuite {

        'fakeTest {
            assert(1 == 1)
        }

    }
}

Console output:

[info] 1/2     net.pointsgame.debug.MyTest.fakeTest     Success
[info] 2/2     net.pointsgame.debug.MyTest      Success
[info] -----------------------------------Results-----------------------------------
[info] 
[info] Tests: 0
[info] Passed: 0
[info] Failed: 0
[success] Total time: 5 s, completed Jun 12, 2014 3:12:29 AM

Were there any recent changes that may have broken the "Results" summary?

Only show the output/error-log-spam once when running a single test.

Showing the same error message three times is dumb; if you're only running one test, you're definitely not going to need to scroll. Running groups of tests it's still valuable to show the three different views (as-it-happens, report, failures) because the size of these groups often differs (e.g. lots of tests, handful of failures).

Combining TestTrees

It would be nice if it were possible to assemble a TestTree from separately defined suites, e.g. something like:

trait TestSpecA {
  val aTests = TestSuite {
    ...
  }
}

trait TestSpecB {
  val bTests = TestSuite {
    ...
  }
}

object Test extends TestSuite with TestSpecA with TestSpecB {
  val tests = aTests ++ bTests
}

normal test names

In scalaspecs2 and in scalatest I can have spaces in test names that makes tests more readable like:

"foo class" should {
"have foo nonempty bar List" in {
//my test
}
//and so on
}

but in u-test I am not allowed to use spaces :_(

Dev tips out of date?

When I run sbt plugins/publishLocal I get:

[error] Expected ID character
[error] Not a valid command: plugins (similar: alias, completions)
[error] Expected project ID
[error] Expected configuration
[error] Expected ':' (if selecting a configuration)
[error] Expected key
[error] Not a valid key: plugins (similar: sbt-plugin, sbtPlugin, publish)
[error] plugins/publishLocal
[error]        ^

SBT issues

When I'm running "test" in SBT, I get an output something like this:

[info] monifu.concurrent.atomic.AtomicAnyTest       Success
[info]     AtomicAny        Success
[info]         set      Failure(utest.AssertionError: (assert(r.get =...

I cannot see the error in the test that failed (the line is truncated) or the line number (don't know if this is possible on Scala.js).

Even more problematic is that the "test" command doesn't report a failure. So in case of continuous integration (with something like Travis-ci), I'm not getting alerts that the test failed:

[info] Tests: 26
[info] Passed: 25
[info] Failed: 1
[success] Total time: 9 s, completed Mar 26, 2014 7:45:19 PM

And here's what happens with ScalaTest (the "test" command fails):

[error] Failed: Total 140, Failed 1, Errors 0, Passed 139
[error] Failed tests:
[error]     monifu.concurrent.atomic.AtomicAnyTest
[error] (monifu/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 22 s, completed Mar 26, 2014 7:50:22 PM

EDIT: for reference, here's the problem manifested in Travis-ci, with the build being reported as successful - https://travis-ci.org/alexandru/monifu/builds/21612155

Added, didn't work, feel sad, JsZip something

" (C:\Users\philip.ivy2\cache\org.webjars\jszip\jars\jszip-2.4.0.jar#META-INF/resources/webjars/jszip/2.4.0/jszip.min.js#12)
at scala.sys.package$.error(package.scala:27)
at org.scalajs.jsenv.rhino.RhinoJSEnv.org$scalajs$jsenv$rhino$RhinoJSEnv$$internalRunJS(RhinoJSEnv.scala:168)
at org.scalajs.jsenv.rhino.RhinoJSEnv$Runner.run(RhinoJSEnv.scala:60)
at org.scalajs.sbtplugin.FrameworkDetector.detect(FrameworkDetector.scala:64)
at org.scalajs.sbtplugin.ScalaJSPluginInternal$$anonfun$40.apply(ScalaJSPluginInternal.scala:453)
at org.scalajs.sbtplugin.ScalaJSPluginInternal$$anonfun$40.apply(ScalaJSPluginInternal.scala:438)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:235)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
error Exception while running JS code: TypeError: Cannot set property "JSZip" of undefined to "
[error] function c(a, b) {
[error] return this instanceof c ? (this.files = {}, this.comment = null, this.root = "", a && this.load(a, b), void (this.clone = funct
ion () {
[error] var a = new c();
[error] for (var b in this) {
[error] "function" != typeof this[b] && (a[b] = this[b]);
[error] }
[error] return a;
[error] })) : new c(a, b);
[error] }
[error] " (C:\Users\philip.ivy2\cache\org.webjars\jszip\jars\jszip-2.4.0.jar#META-INF/resources/webjars/jszip/2.4.0/jszip.min.js#12)
[error] Total time: 3 s, completed Feb 24, 2015 5:42:38 PM

C:\home\projects\shape\workbench-example-app-master>

Compile error when calling a local def inside the assert macro

See this test case, which fails to compile because, inside the assert macro, I call a local def:
sjrd@e834c79
repeated here fore convenience:

'applyLocalDefInAssert{
  def len(str: String) = str.length
  val a = "scala rocks"
  assert(len(a) == 11)
}

The compile error is:

[error] C:\Users\Sepi\Documents\Projets\utest\jvm\..\shared\test\scala\utest\Asserts.scala:89: missing arguments for method len;
[error] follow this method with `_' if you want to treat it as a partially applied function
[error]         assert(len(a) == 11)
[error]                ^

The cause is probably that it tries to display the value of len (because it is an identifier mentioned in the asserted expression) but of course this does not compile because len is not a value, it is a local def.

java.lang.IllegalArgumentException: Could not find proxy

This compiles and runs as expected:

"the comment" - { assert(true) }

but the following

val comment = "the comment" 
comment - { assert(true) }

fails at compile time with an error that look like that:

[error] (ldpatchJVM/test:compile) java.lang.IllegalArgumentException: Could not find proxy for val comment: String in List(value comment, method positiveEvaluationTest, method tests, class LDPatchTestSuite, package ldpatch, package banana, package w3, package org, package <root>) (currentOwner= method tests )

I need to generate tests from external files, so their names are not known at compile time. Is that a limitation of utest?

print matched value for assertMatch

When I do assertMatch, make sbt test and it fails I get something like " Failure(utest.AssertionError: (assertMatch(new TestTermParser("http://google.com.ua").IRI_REF.run()){case Success(IRI("http://google.com.ua"))=>}"

But I do not see the actual value of matched variable. That is why I have to add additional lines to see the real value before assertMatch when test fails. It would be great (and would save some time) if together with assertion failure it also printed toString value of the variable.

sbt not running tests

I've followed the tutorial and the utest README but sbt seems to ignore the tests in my (really simple) project https://github.com/fommil/kerbal-calculator/

Can you please take a look and see what I'm doing wrong? It might lead to a change to the docs.

Testsuite hierarchy leads to weirdly formatted report

Just trying out uTest and running into an issue. When I have generic tests in a base class and specific tests in a subclass--together forming a TestSuite, running uTest from sbt leads to a weirdly formatted report.

Here is the code to reproduce (In the real scenario, the base class is parameterized, but keeping things simple here):

package example

import utest._
import utest.framework.{Test, TestSuite}
import utest.util.Tree

abstract class CommonTest extends TestSuite {
  val specificTests : Tree[Test]

  val tests = TestSuite {
    "common" - {
      assert(true)
    }

    "specific" - {
      specificTests
    }
  }
}

object SpecificTest extends CommonTest {
  override val specificTests = TestSuite {
    "specific 1" - {
      assert(true)
    }

    "specific 2" - {
      assert(true)
    }
  }
}

SBT output:

> test
[info] 1/3     example.SpecificTest.common      Success
[info] 2/3     example.SpecificTest.specific        Success(Tree(Test(example.SpecificTest,utest.framework.TestThunkTree@60c7b31a),ArrayBuffer(Tree(Test(specific 1,utest.framework.TestThunkTree@60c7b31a),List()), Tree(Test(specific 2,utest.framework.TestThunkTree@60c7b31a),List()))))
[info] 3/3     example.SpecificTest     Success
[info] -----------------------------------Results-----------------------------------
[info] example.SpecificTest     Success
[info]     common       Success
[info]     specific     Success(Tree(Test(example.SpecificTest,utest.framework.TestThunkTree@60c7b31a),ArrayBuffer(Tree(Test(specific 1,utest.framework.TestThunkTree@60c7b31a),List()), Tree(Test(specific 2,utest.framework.TestThunkTree@60c7b31a),List()))))
[info] Failures:
[info] 
[info] Tests: 3
[info] Passed: 3
[info] Failed: 0
[success] Total time: 0 s, completed Nov 2, 2014 2:04:58 PM

Add asynchronous support for ScalaJS please

Considering the fact that in JS stuff is usually asynchronous, it is really tough to go around without asynchronously thrown AssertionErrors :

setTimeout( () => {assert(condition)},123)

being reflected/included into test results or at least making the entire test suite fail. Currently they are utterly invisible...

If this feature request is not adequate, would you please give us advice about the best way to approach this problem? I can think of a few ways of doing that, but they are all rather ugly unreliable workarounds.

environment: PhantomJS with autoExit=false

AssertionFailed: utest.desiredOutput != utest.actualOutput

I've used utest pretty extensively to develop and debug Prickle.

Plus side: cross JS/JVM testing is the killer feature, and also I trust that utest basically "works". But the actual test output is miles from what I'd ideally like, it often leaves me cursing when Im working with it, and I feel a bunch of defaults could be changed for a much better UX.

This issue going to be a case study in (a) what's there now, (b) what I'd like, and (c) what changes would be needed to close the gap. There will be overlap with some other existing issues. Over coming weeks I have some time free to help with PRs to move it in the direction I'd like to see it go.

(a) Actual Output. Prickle, with one test failing:

> jvm/test
utest.asserts.package$.assertError(package.scala:19)
utest.asserts.package$AssertEntryExt$$anonfun$get$1.apply(package.scala:38)
utest.asserts.package$AssertEntryExt$$anonfun$get$1.apply(package.scala:38)
utest.asserts.Asserts$$anonfun$assertImpl$1.apply(Asserts.scala:72)
utest.asserts.Asserts$$anonfun$assertImpl$1.apply(Asserts.scala:70)
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
utest.asserts.Asserts$.assertImpl(Asserts.scala:70)
prickle.PickleTests$$anonfun$tests$36$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$23.apply(PicklerTest.scala:37)
prickle.PickleTests$$anonfun$tests$36$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$23.apply(PicklerTest.scala:17)
[info] 1/36    prickle.PickleTests.with cyclic config.caseclass.encoding        Success
[info] 2/36    prickle.PickleTests.with cyclic config.caseclass.unpickling      Success
[info] 3/36    prickle.PickleTests.with cyclic config.caseclass.toleratesextradata      Failure(utest.AssertionError: assert(benDetails != actual)
[info] actual: prickle.PersonalDetails = PersonalDetails(Person(Ben),null,40,false,175.6667,Person(Keith),AnObject,ModelNumber(V,2000,42,1234567890,-0.125,365 days)))
[info] 4/36    prickle.PickleTests.with cyclic config.caseclass     Success
[info] 5/36    prickle.PickleTests.with cyclic config.generic       Success
[info] 6/36    prickle.PickleTests.with cyclic config.compositepicklers.apple.pickle        Success
[info] 7/36    prickle.PickleTests.with cyclic config.compositepicklers.apple.unpickle      Success
[info] 8/36    prickle.PickleTests.with cyclic config.compositepicklers.apple       Success
[info] 9/36    prickle.PickleTests.with cyclic config.compositepicklers.carrot.pickle       Success
[info] 10/36   prickle.PickleTests.with cyclic config.compositepicklers.carrot.unpickle     Success
[info] 11/36   prickle.PickleTests.with cyclic config.compositepicklers.carrot      Success
[info] 12/36   prickle.PickleTests.with cyclic config.compositepicklers.null.pickle     Success
[info] 13/36   prickle.PickleTests.with cyclic config.compositepicklers.null.unpickle       Success
[info] 14/36   prickle.PickleTests.with cyclic config.compositepicklers.null        Success
[info] 15/36   prickle.PickleTests.with cyclic config.compositepicklers     Success
[info] 16/36   prickle.PickleTests.with cyclic config.maps      Success
[info] 17/36   prickle.PickleTests.with cyclic config.sorted maps       Success
[info] 18/36   prickle.PickleTests.with cyclic config.dates     Success
[info] 19/36   prickle.PickleTests.with cyclic config.seqs.shared object support        Success
[info] 20/36   prickle.PickleTests.with cyclic config.seqs      Success
[info] 21/36   prickle.PickleTests.with cyclic config.iterable      Success
[info] 22/36   prickle.PickleTests.with cyclic config.sets.shared object support        Success
[info] 23/36   prickle.PickleTests.with cyclic config.sets      Success
[info] 24/36   prickle.PickleTests.with cyclic config.tuple shared object support       Success
[info] 25/36   prickle.PickleTests.with cyclic config.case_class_with_doubly_nested_parametric_field        Success(Success(DoublyNested(None)))
[info] 26/36   prickle.PickleTests.with cyclic config.intoString        Success
[info] 27/36   prickle.PickleTests.with cyclic config.fromString        Success
[info] 28/36   prickle.PickleTests.with cyclic config       Success
[info] 29/36   prickle.PickleTests.with non-shared config.over shared structure.shared structures are duplicated        Success
[info] 30/36   prickle.PickleTests.with non-shared config.over shared structure.data is preserved       Success
[info] 31/36   prickle.PickleTests.with non-shared config.over shared structure.no id tags added to pickle      Success
[info] 32/36   prickle.PickleTests.with non-shared config.over shared structure.no state during pickle      Success
[info] 33/36   prickle.PickleTests.with non-shared config.over shared structure.no state during unpickle        Success
[info] 34/36   prickle.PickleTests.with non-shared config.over shared structure     Success
[info] 35/36   prickle.PickleTests.with non-shared config       Success
[info] 36/36   prickle.PickleTests      Success
[info] -----------------------------------Results-----------------------------------
[info] prickle.PickleTests      Success
[info]     with cyclic config       Success
[info]         caseclass        Success
[info]             encoding     Success
[info]             unpickling       Success
[info]             toleratesextradata       Failure(utest.AssertionError: assert(benDetails != actual)
[info]             actual: prickle.PersonalDetails = PersonalDetails(Person(Ben),null,40,false,175.6667,Person(Keith),AnObject,ModelNumber(V,2000,42,1234567890,-0.125,365 days)))
[info]         generic      Success
[info]         compositepicklers        Success
[info]             apple        Success
[info]                 pickle       Success
[info]                 unpickle     Success
[info]             carrot       Success
[info]                 pickle       Success
[info]                 unpickle     Success
[info]             null     Success
[info]                 pickle       Success
[info]                 unpickle     Success
[info]         maps     Success
[info]         sorted maps      Success
[info]         dates        Success
[info]         seqs     Success
[info]             shared object support        Success
[info]         iterable     Success
[info]         sets     Success
[info]             shared object support        Success
[info]         tuple shared object support      Success
[info]         case_class_with_doubly_nested_parametric_field       Success(Success(DoublyNested(None)))
[info]         intoString       Success
[info]         fromString       Success
[info]     with non-shared config       Success
[info]         over shared structure        Success
[info]             shared structures are duplicated     Success
[info]             data is preserved        Success
[info]             no id tags added to pickle       Success
[info]             no state during pickle       Success
[info]             no state during unpickle     Success
[info] Failures:
[info] 3/36    prickle.PickleTests.with cyclic config.caseclass.toleratesextradata      Failure(utest.AssertionError: assert(benDetails != actual)
[info] actual: prickle.PersonalDetails = PersonalDetails(Person(Ben),null,40,false,175.6667,Person(Keith),AnObject,ModelNumber(V,2000,42,1234567890,-0.125,365 days)))
[info] Tests: 36
[info] Passed: 35
[info] Failed: 1
[success] Total time: 0 s, completed Nov 16, 2014 9:38:11 PM

Desired Default output for above situation:

> jvm/test
[info] Passed
[info] prickle.PickleTests      Success
[info]     with cyclic config       Success
[info]         caseclass        Success
[info]             encoding     Success
[info]             unpickling       Success
[info]         generic      Success
[info]         compositepicklers        Success
[info]             apple        Success
[info]                 pickle       Success
[info]                 unpickle     Success
[info]             carrot       Success
[info]                 pickle       Success
[info]                 unpickle     Success
[info]             null     Success
[info]                 pickle       Success
[info]                 unpickle     Success
[info]         maps     Success
[info]         sorted maps      Success
[info]         dates        Success
[info]         seqs     Success
[info]             shared object support        Success
[info]         iterable     Success
[info]         sets     Success
[info]             shared object support        Success
[info]         tuple shared object support      Success
[info]         case_class_with_doubly_nested_parametric_field       Success
[info]         intoString       Success
[info]         fromString       Success
[info]     with non-shared config       Success
[info]         over shared structure        Success
[info]             shared structures are duplicated     Success
[info]             data is preserved        Success
[info]             no id tags added to pickle       Success
[info]             no state during pickle       Success
[info]             no state during unpickle     Success
[info] Failures:
[info] 3/36    prickle.PickleTests.with cyclic config.caseclass.toleratesextradata      Failure(utest.AssertionError: assert(benDetails != actual)
utest.asserts.package$.assertError(package.scala:19)
utest.asserts.package$AssertEntryExt$$anonfun$get$1.apply(package.scala:38)
utest.asserts.package$AssertEntryExt$$anonfun$get$1.apply(package.scala:38)
utest.asserts.Asserts$$anonfun$assertImpl$1.apply(Asserts.scala:72)
utest.asserts.Asserts$$anonfun$assertImpl$1.apply(Asserts.scala:70)
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
utest.asserts.Asserts$.assertImpl(Asserts.scala:70)
prickle.PickleTests$$anonfun$tests$36$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$23.apply(PicklerTest.scala:37)
prickle.PickleTests$$anonfun$tests$36$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$23.apply(PicklerTest.scala:17)
[info] actual: prickle.PersonalDetails = PersonalDetails(Person(Ben),null,40,false,175.6667,Person(Keith),AnObject,ModelNumber(V,2000,42,1234567890,-0.125,365 days)))
[info] benDetails: prickle.PersonalDetails = PersonalDetails(Person(Ben),null,40,false,175.6667,Person(Keith),AnObject,ModelNumber(V,2000,42,1234567890,-0.125,365 days)))
[info] Tests: 36, Passed: 35, Failed: 1
[success] Total time: 0 s, completed Nov 16, 2014 9:38:11 PM

Reducing verbosity. Both screen space and mental attention are scarce in programming. Right now, utest emits more output by default than is useful. In (b), I've only reported the output of each test exactly once, in either a Pass or Failed section. I've rolled up the final counts to a single line. Ive dropped the --- Results --... line as largely self-explanatory.
Stack traces, once in the right spot, underneath the associated Exception in the Failures.
All operands to failed asserts listed. In my case study, the failing assert compares a local variable (actual) against a field (benDetails). But the utest output only prints the local variable. Similarly, anonymous expressions within an assert() go unreported as well.

Printing the values of failed equality checks is core responsibility of a test framework. Its no good having fancy macro-powered asserts, unless they can do what def assertEquals(expected, actual) method can do and print both sides upon failure.

A minor tweak. In test 25/36 only, success is reported as Success(Success(DoublyNested(None))). It's less distracting if its reported identically to all the others, just Success

With all these points, interested to hear to what extent you agree in principle, putting aside for a moment questions of the work required to implement them.

Intercept doesn't work with variable assignments

Probably not a huge deal, but there's no reason it shouldn't work

var W = 1
intercept[Exception] {
  W = 2
}
[error] Test.scala:88: assignment to non variable
[error]             W = 2
[error]               ^

Add should matchers

In most of the cases should matchers are more convenient than asserts.
It would be great to have some should matchers like in all normal (scalatest,scalaspecs2 and so on) testing frameworks

foo.bar should beTrue
foo.bar shouldEqual foo2.bar

uTest interacts badly with Angular.js?

I cannot run tests with angularjs as dependency:

jsDependencies ++= Seq (
"org.webjars" % "angularjs" % "1.3.14" / "angular-route.js" dependsOn "angular.js",
"org.webjars" % "angularjs" % "1.3.14" / "angular.js"
)

The problem returned is org.scalajs.jsenv.ExternalJSEnv$NonZeroExitException: PhantomJS exited with code 2

When using without jsDependencies += RuntimeDOM the nodejs process return exit 1. Comment the dependencies work and the test run

Prettier assertions

The current set of smart asserts is workable, but it would be nice to have even prettier ones like expecty or now scalatest

It's harder than it looks, since the trees that are available to the macros are already greatly transformed and look nothing like the source code, the lack of unprinting facilities/rangepositions, and the difficulty in extracting an existing open source implementation (e.g. one of those above) and merging it in.

Definitely doable though if anyone is looking for a way to contribute!

Crashes on import inside TestSuite block

object Bippy extends TestSuite {
  val tests = TestSuite {
    import Predef._
    "bar" - assert(true)
  }
}

leads to

scala.reflect.internal.FatalError:
  unexpected tree: class scala.reflect.internal.Trees$Import
import scala.Predef._
        during phase: globalPhase=typer, enteringPhase=namer
     library version: version 2.11.0-M8
    compiler version: version 2.11.0-M8
  last tree to typer: Import(value <import>)
       tree position: line 34 of /r/psp-view/src/test/scala/NatSpec.scala
              symbol: value <import>
   symbol definition: val <import>: ImportType(scala.Predef) (a TermSymbol)
      symbol package: <none>
       symbol owners: value <import>
           call site: object BippySpec in package tests in package tests

fastOptStage::test not running on phantomJS when using utest

Hi! I am not sure if this is a bug or if I'm missing something. When I test my project, a test involving an ArrayBuffer fails with Failure(TypeError: org.mozilla.javascript.Undefined@62042b18 is not a function, it is undefined.). My build.sbt is configured like this:

scalaJSSettings

name := "My Project"

scalaVersion := "2.11.1"

libraryDependencies += "org.scala-lang.modules.scalajs" %%% "scalajs-dom" % "0.6"

libraryDependencies += "com.lihaoyi" %%% "utest" % "0.1.6" % "test"

ScalaJSKeys.requiresDOM := true

ScalaJSKeys.scalaJSTestFramework := "utest.jsrunner.JsFramework"

(loadedTestFrameworks in Test) := {
  (loadedTestFrameworks in Test).value.updated(
    sbt.TestFramework(classOf[utest.jsrunner.JsFramework].getName),
    new utest.jsrunner.JsFramework(environment = (ScalaJSKeys.jsEnv in Test).value)
  )
}

testLoader := scala.scalajs.sbtplugin.testing.JSClasspathLoader((ScalaJSKeys.execClasspath in Compile).value)

I've tested the same thing with JasmineTestRunner and it works when run with fastOptStage::test but fails with the same type error when run just with test. My guess is that utest's configuration somehow prevents fastOptStage::test from running with phantomJS and instead runs it with rhino. Is some extra configuration missing? Thanks in advance.

sjs version does not work

In my project ( https://github.com/antonkulaga/semweb ) scalaJS version of tests does not work, I have numerious errors like

TypeError: Cannot read property "rdf" from undefined

failures not reported

This is the summary of my test run:

[info] Tests: 5
[info] Passed: 4
[info] Failed: 1
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0

It looks like the aggregate is ignoring the individual scores.

macro runner

Hey!

what's your status on this ?
https://groups.google.com/forum/#!topic/scala-user/gSZ7WDdbb6k

Debug of variables

It would be nice to have a way to debug variables, println do not work.
In javascript land, you constant doens't know the content of variable or object until you look at. The browser ofers the console.debug that print a nice representation of any object.

I'm constant struggling to test things like that:

utest.AssertionError: assert(ndom.children().length == 0);
[info]                                              ndom: org.scalajs.jquery.JQuery = [object Object]

Why this assertion don't work? The length of the children is a object instead of Int? What is the value?
One quick and dirty way is to log the the output of lenght and see what it has but it is not possible to do that.

The most strange thing is that this test pass sometimes, so the problem here is that utest is unable to provide the content of the variable in the test result?

com-lihaoyi / utest Goto Github PK

utest's People

Contributors

Stargazers

Watchers

Forkers

utest's Issues

Recommend Projects

Recommend Topics

Recommend Org