anotherchrisberry / spock-retry Goto Github PK

View Code? Open in Web Editor NEW

34.0 34.0 15.0 51 KB

Spock extension to allow test features and specifications to be retried before failed

License: MIT License

Groovy 100.00%

spock-retry's People

Contributors

Stargazers

Watchers

Forkers

nr9 levsa agdula strido littletonman pennstatephil vishnuvenu jroyg cybernetics travisspencer iousin timetraveler90 dhc8 rkrdpm henrik242

spock-retry's Issues

Not compatible with latest Spock core version

My project uses Spock core 1.0-groovy-2.4
If I install spock-retry as is,
I get

groovy.lang.MissingPropertyException: No such property: cleanupMethod for class: org.spockframework.runtime.model.SpecInfo
Possible solutions: cleanupMethods
    at com.anotherchrisberry.spock.extensions.retry.RetryInterceptor.intercept(RetryInterceptor.groovy:26)
    at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:87)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)

So I cloned and tried to change the dependency by myself. In that case it fails to build

Specs with @Stepwise fail as retry all tests in spec, only executes failed tests

I like this extension a lot and we use it in our daily testing but we had a strange situation where we had to execute tests sequentially so, we used @stepwise to accomplish that.

We had one test which was failing inside spec with @stepwise and retry executed that specific test. In this situation I would expect @RetryOnFailure to execute all tests inside that spec instead of only failed tests as it is annotated with @stepwise

Makes @PendingFeature not work

@PendingFeature is useful for marking tests as skipped if they fail due to a known bug (or missing feature). Unfortunately, if a feature is marked with both @PendingFeature and @RetryOnFailure, it will fail instead of being marked skipped.

It looks like this is because the pending feature annotation is caught as the first retry, and then subsequent retries aren't handled correctly by it:

13:06:54,664 Retry caught failure 1 / 2: 
org.junit.AssumptionViolatedException: Feature not yet implemented correctly.
	at org.spockframework.runtime.extension.builtin.PendingFeatureBaseInterceptor.assumptionViolation(PendingFeatureBaseInterceptor.java:28)
	at org.spockframework.runtime.extension.builtin.PendingFeatureInterceptor.intercept(PendingFeatureInterceptor.java:20)
	at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:87)
	at org.spockframework.runtime.extension.IMethodInvocation$proceed.call(Unknown Source)
	at com.anotherchrisberry.spock.extensions.retry.RetryInterceptor.intercept(RetryInterceptor.groovy:28)
	at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:87)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:472)
	at org.spockframework.runtime.BaseSpecRunner.runFeatureMethod(BaseSpecRunner.java:406)
	at org.spockframework.runtime.BaseSpecRunner.doRunIteration(BaseSpecRunner.java:324)
	at org.spockframework.runtime.BaseSpecRunner$6.invoke(BaseSpecRunner.java:309)
	at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:481)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:464)
	at org.spockframework.runtime.BaseSpecRunner.runIteration(BaseSpecRunner.java:288)
	at org.spockframework.runtime.BaseSpecRunner.initializeAndRunIteration(BaseSpecRunner.java:278)
	at org.spockframework.runtime.BaseSpecRunner.runSimpleFeature(BaseSpecRunner.java:269)
	at org.spockframework.runtime.BaseSpecRunner.doRunFeature(BaseSpecRunner.java:263)
	at org.spockframework.runtime.BaseSpecRunner$5.invoke(BaseSpecRunner.java:246)
	at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:481)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:464)
	at org.spockframework.runtime.BaseSpecRunner.runFeature(BaseSpecRunner.java:238)
	at org.spockframework.runtime.BaseSpecRunner.runFeatures(BaseSpecRunner.java:188)
	at org.spockframework.runtime.BaseSpecRunner.doRunSpec(BaseSpecRunner.java:98)
	at org.spockframework.runtime.BaseSpecRunner$1.invoke(BaseSpecRunner.java:84)
	at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:481)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:464)
	at org.spockframework.runtime.BaseSpecRunner.runSpec(BaseSpecRunner.java:76)
	at org.spockframework.runtime.BaseSpecRunner.run(BaseSpecRunner.java:67)
	at org.spockframework.runtime.Sputnik.run(Sputnik.java:63)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at sun.reflect.GeneratedMethodAccessor345.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at com.sun.proxy.$Proxy3.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:108)
	at sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:146)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:128)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
	at java.lang.Thread.run(Thread.java:745)
13:06:54,779 Retry caught failure 2 / 2: 
Condition not satisfied:
<snip - real test failure here>

Latest release (0.6.5) needs to be deployed to jcenter/bintray

Was there a reason why the latest release was not deployed to bintray as described in the README?

Tests are not rerun if failure because of WaitTimeoutException

Is it correct that timeout failures aren't caught by this extension and retried? I get a test that fails with

geb.waiting.WaitTimeoutException: condition did not pass in 20.0 seconds (failed with exception)

which does not cause reruns.

AssumptionViolatedException should not cause a retry

If a test throws a org.junit.AssumptionViolatedException, a retry should not be triggered. That is used to say the test should be ignored. Retrying it under those conditions isn't necessary.

Invoke custom logic on failure

HI Chris.

I'm trying to add some custom logic that would be invoked after failure besides cleanupMethods and setupMethods. invocation.target.invokeMethod("some method name") in void intercept(IMethodInvocation invocation) throws Throwable does not seem to be working. You probably have this figured out so if it's not a problem tell me how to invoke a custom method on a Spec.

I need this to invoke some code that will do some cleanup that does not need to be run in normal cases.

java.lang.NoClassDefFoundError: com/anotherchrisberry/spock/extensions/retry/RetryInterceptor$_intercept_closure1

Any ideas why this error keeps happening?

java.lang.NoClassDefFoundError: com/anotherchrisberry/spock/extensions/retry/RetryInterceptor$_intercept_closure1

    at com.anotherchrisberry.spock.extensions.retry.RetryInterceptor.intercept(RetryInterceptor.groovy:41)
    at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:87)
    at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:88)
    at org.spockframework.runtime.extension.builtin.AbstractRuleInterceptor$1.evaluate(AbstractRuleInterceptor.java:37)
    at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
    at org.spockframework.runtime.extension.builtin.TestRuleInterceptor.intercept(TestRuleInterceptor.java:38)
    at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:87)
    at org.junit.runners.Suite.runChild(Suite.java:128)
    at org.junit.runners.Suite.runChild(Suite.java:27)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:262)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.lang.ClassNotFoundException: com.anotherchrisberry.spock.extensions.retry.RetryInterceptor$_intercept_closure1
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 21 more

update Spock version

I have a project that was updated to use spock-core:1.1-groovy-2.4, which is causing errors while running tests due to 2 versions of the same library (since this extension uses 1.0-groovy-2.4). I'll submit a PR with a version bump to fix it.

java.lang.NoClassDefFoundError

Any ideas how to fix the below error? I copied all the files here into my project, but I sometimes get this error when running my tests:

java.lang.NoClassDefFoundError: cchproj/orea/park/apitests/utils/RetryInterceptor$_intercept_closure1

	at cchproj.orea.park.apitests.utils.RetryInterceptor.intercept(RetryInterceptor.groovy:41)
	at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:87)
	at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:88)
	at org.spockframework.runtime.extension.builtin.AbstractRuleInterceptor$1.evaluate(AbstractRuleInterceptor.java:37)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.spockframework.runtime.extension.builtin.TestRuleInterceptor.intercept(TestRuleInterceptor.java:38)
	at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:87)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:262)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.lang.ClassNotFoundException: cchproj.orea.park.apitests.utils.RetryInterceptor$_intercept_closure1
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 13 more

Project is not on JCenter

Hi!

I would really love to use this but you don't seem to have published the artifacts on JCenter (or Maven Central) so it's hard to get your library automatically with Maven/Gradle.

Would you be able to push this to JCenter? As you already have the project on Bintray, it should be trivial to do that.

This page explains how to do it.

Regards.

Exception thrown in setup mark test as success even if it failed

Let me present scenario:

package no.spid.payment.acceptance.milestone.external;

import com.anotherchrisberry.spock.extensions.retry.RetryOnFailure;
import spock.lang.Specification;

@RetryOnFailure
class RetryTest extends Specification {

    def setup() {
        throw new RuntimeException("setup")
    }

    def test() {
        given:
        throw new RuntimeException("test")
    }
}

This test ends with success, because after exception is caught during test execution then RetryInterceptor is invoking setup method again but when exception is caught it's only increasing attempts which leads to marking test as success cause it isn't trying to execute test again and rethrow exception, because attempts <= retryMax is false.

spock-retry is affected by CVE-2020-17521 because of old groovy dependency

See

Wait time between retries

Would love to see an option like waitSeconds=5 to wait a configurable number of seconds between tries (so if you're rate limited, for example, you can slow down the requests)