sbabcoc / junit-foundation Goto Github PK

Hello there.

I've tried to update our library on JUnit-Foundation version 16 and it look like it does not support JunitParams library anymore. All my tests related to this functionality fail. From what I can see no testStarted and beforeInvocation calls for such tests.

At present, the unit test suite for JUnit Foundation lacks explicit verification that matching start/finish notifications are balanced and fired in the correct sequence. This is a serious omission, given that these notifications are the raison d'être for this project.

As Kotlin provides to use multi-word method names, it is make incompatible to use this library with Kotlin's tests which use such methods.

There is an explicit splitting of test names inside of AtomicTest class constructor. Providing multi-word method name, this constructor takes only the first word of this method's name and tries to find this name in given particles. Of course it will find nothing.

Steps to reproduce:

1. Create Kotlin class.
2. Create test method using multi-words syntax (for example fun 'some method'() {...})
3. Integrate it with JUnit-Foundation and run test

Expected behavior:

AtomicTest class constructor initializes identity field with created method.

Actual behavior:

AtomicTest class constructor does not initializes identity field.

... or at least confirm that it works as-is.

The project repository can be found here.

Using JUnitParams for running parameterized tests fail with NullPointerException.
To reproduce the issue you can use next branch
Just deploy RP locally and run mvn test to start com.epam.reportportal.example.junit.logging.JUnitParamsTest or use running via Junit plugin for more detailed stacktrace

15:30:30.079 DEBUG c.n.a.j.RunReflectiveCall - main - beforeInvocation: logPlain(17, false) 0
15:30:30.128 DEBUG c.n.a.j.RunReflectiveCall - main - afterInvocation: logPlain(17, false) 0
15:30:30.021 DEBUG c.n.a.j.RunAnnouncer - main - testStarted: logPlain(17, false) 0
15:30:30.165 DEBUG c.n.a.j.RunAnnouncer - main - testStarted: logPlain(22, true) 1
15:30:30.168 DEBUG c.n.a.j.RunReflectiveCall - main - beforeInvocation: logPlain(22, true) 1
15:30:30.169 DEBUG c.n.a.j.RunReflectiveCall - main - afterInvocation: logPlain(22, true) 1
15:30:30.158 DEBUG c.n.a.j.RunAnnouncer - main - testFinished: logPlain(17, false) 0
15:30:30.164 DEBUG c.n.a.j.CreateTest - main - testObjectCreated: com.epam.reportportal.example.junit.logging.JUnitParamsTest@7645b7d
15:30:30.171 DEBUG c.n.a.j.RunAnnouncer - main - testFinished: logPlain(22, true) 1
15:30:30.179 DEBUG c.n.a.j.Run - main - runFinished: junitparams.JUnitParamsRunner@45eab322

Process finished with exit code 255

java.lang.NullPointerException
at com.nordstrom.automation.junit.RunAnnouncer.isSupported(RunAnnouncer.java:144)
at com.nordstrom.automation.junit.RunAnnouncer.testStarted(RunAnnouncer.java:27)
at org.junit.runner.notification.SynchronizedRunListener.testStarted(SynchronizedRunListener.java:49)
at org.junit.runner.notification.RunNotifier$3.notifyListener(RunNotifier.java:121)
at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72)
at org.junit.runner.notification.RunNotifier.fireTestStarted(RunNotifier.java:118)
at org.junit.internal.runners.model.EachTestNotifier.fireTestStarted(EachTestNotifier.java:42)
at junitparams.internal.ParameterisedTestMethodRunner.runMethodInvoker(ParameterisedTestMethodRunner.java:45)

To indicate that timeouts should be disabled (both Timeout rule and annotation element), set TIMEOUT_RULE to 0 or declare a Timeout rule in the class specifying 0 as the timeout.

Note that this description was copied from scribbled notes. It will probably be necessary to crack open the related code to truly understand what it means.

Description doesn't fully resolve for JUnitParams until Parameterized...Runner.runTestMethod(). Only JUnitParams needs Runnable.
EachTestNotifier constructor is the first core method to get the fully resolved Description.

• Intercept EachTestNotifier.fireTestStarted() and EachTestNotifier.fireTestFinished().
• Every Statement includes the target object.
• Intercept the methodBlock() method. This has runner and method, which are unique enough to work if values are collected as a queue. In the EachTestNotifier intercept, map from Description to target, draining the **MethodBlock queue.
• InvokeMethod is guaranteed to have the target.

Some example code in README uses old method signatures. This code needs to be updated to use the current method signatures.

Hi @sbabcoc , I am having issues with attempting to attach my JUnit RunListener in my gradle project. I have followed all of the steps outlined in the article here -> https://discuss.gradle.org/t/how-to-attach-a-runlistener-to-your-junit-4-tests-in-gradle/30788 and my project is still not picking up the RunListener. Any help would be appreciated, some project specs:

• Geb/Spock Java/Groovy project
• Headless Chrome tests
• JUnit 4
• Attempting to attach a the com.adaptavist.tm4j.junit.ExecutionListener provided by TM4J for JIRA

Specific issue is that there is no indication that the RunListener is being picked up at execution time, and no contextual logging is being output that would indicate errors/misuse

Currently, RunListener providers can only be declared in their native SPI configuration file. Revise the service loader to recognize providers declared in the JUnitWatcher SPI configuration file, too.

You are using service loader to load customer listeners and watchers by component (one listener - one service file). It makes sense to create some marker interface (like LifecycleItem) and load all listeners and watchers on one ServiceLoader call. You'l resolve and collect our listeners by type on your side but users will be have only one META-INF/services/LifecycleItem file in the classpath.

Port this back from JUnit 5

JUnit 5 provides a TestExecutionExceptionHandler interface that can be used to define the behavior of a test when encountering certain types of exceptions. This is a useful feature that should be fairly easy to implement.

Enabling automatic retry with tests that are parameterized with the Theories runner results in bad behavior. The solution is to factor out the core of RetryHandler.runChildWithRetry(), passing in the statement that gets stashed away by the "theory catalyst" of the MethodBlock class. This will fix the incompatibility, and will also eliminate a bunch of redundant code in the RunWithCompleteAssignment class.

Note that the RunChild interceptor must be cognizant of the Theories runner, foregoing redirection of execution to the retry handle at this point to avoid attempting to retry the entire set of method permutations.

Hi,
Currently, this library is overriding timeout annotation to have a global timeout.
Junit has deprecated this option and introduced Timeout Rule, which also considers the time spent in the before and after method along with the unit test's time. (timeout annotation doesn't consider before and after method time).
I've made this change in the library via overriding getTestRules method of BlockJUnit4ClassRunner and adding timeout rule to existing rule list.
Please share your thought on same.

Thanks

These references are collected in the WATCHER_MAP static field. These should be released at the end of each atomic test to avoid the accumulation of objects that can exhaust available memory.

Hey, not sure if there is a planned change already but I've encountered an issue with the LifecycleHooks class in the installTransformer method.
If I try and run my JUnit tests passing the junit-foundation-11.0.1.jar as a javaagent i'm getting the following error:
Caused by: java.lang.NoSuchMethodError: net.bytebuddy.pool.TypePool$Default.ofSystemLoader()Lnet/bytebuddy/pool/TypePool; at com.nordstrom.automation.junit.LifecycleHooks.installTransformer(LifecycleHooks.java:72) at com.nordstrom.automation.junit.LifecycleHooks.premain(LifecycleHooks.java:62)

Looks like that ofSystemLoader() currently can't be found.

I've come across this while trying to get an instance of ReportPortal working and my delving lead me to find this. Whenever I tried running via Maven Surefire I was getting the following error in the dumpstream:
Corrupted STDOUT by directly writing to native stream in forked JVM 1. Stream 'FATAL ERROR in native method: processing of -javaagent failed'. java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'FATAL ERROR in native method: processing of -javaagent failed'. at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:507) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:210) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:177) at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88) at java.base/java.lang.Thread.run(Thread.java:835)

So i tried tracing down to see if it was surefire or an issue in the code. Based on the description provided in ReportPortal documentation "using IDE Run configurations" I supplied the Javaagent in the VM options and identified an issue in this method call.

Here's a snippet of what happens when I try running a basic test in IntelliJ with the javaagent set in the IDE run configuration (don't mind the extra colours, red's still the bad guy here):

For reference I'm using Maven v5.3.4, JDK 12.0.2, junit-foundation-11.0.1.jar (and when running via maven, surefire v3.0.0-M3). Not sure if this could have had an impact but if it does please let me know.

While upgrading the JUnit Foundation dependency of agent-java-junit, I discovered that we couldn't handle suite-level (@BeforeClass) failures. The notifications generated for such failures are associated with the class, not a specific JUnit test of the class. However, the AtomicTest class had no way to represent the description for a JUnit class.
I modified this class so it can handle non-test elements, although this is definitely not what clients are likely to expect. These "atomic non-tests" have no identity and no particle methods. I could potentially attach the specific configuration method that caused the failure, but this may only lead to confusion.
At the least, I need to add at least one unit test to verify that suite-level configuration method failures are handled properly.

Currently, the only types of watchers that will be exported are MethodWatcher objects. None of the other watcher types are available. This is a huge hole versus the support provided by TestNG Foundation.

Currently, the MutableTest class uses reflection to access the list of declared annotations of the Java Executable class. The objective is to replace the standard Test annotation with a mutable proxy to enable overriding the value of the timeout field.
Reflective access to the required private field of the Executable class is no longer legal. This deprecated functionality can be permitted with the --illegal-access=permit option, but this is definitely sub-optimal. This illegal implementation should be replaced with supported functionality implemented with the Advice feature of Byte Buddy.

An example of this sort of implementation can be found here.
A comprehensive guide can be found here.
To intercept private methods and access private fields, check this.
The MemberSubstitution unit tests can be found here.
Example of attaching advice to a core Java class is found here.

The theory is that we can intercept the execution of Executable.declaredAnnotations() on exit, replacing the standard Test annotation with an instance of MutableTest. The code in MockMethodAdvice of Mockito is seemingly applicable to this scenario.

Despite the fact that it's marked as 'experimental' 'Theories' framework is a pretty old feature. I can track it up to JUnit 4.6, the earliest version available on github. But if I try to run even a simplest test with @Theory annotation JUnit Foundation fails with a NullPointerException:

java.lang.NullPointerException
	at com.nordstrom.automation.junit.TimeoutUtils.applyTestTimeout(TimeoutUtils.java:93)
	at com.nordstrom.automation.junit.CreateTest.intercept(CreateTest.java:38)
	at org.junit.experimental.theories.Theories$TheoryAnchor$1.createTest(Theories.java)
	at org.junit.runners.BlockJUnit4ClassRunner.createTest$original$16gso2gv(BlockJUnit4ClassRunner.java:260)
	at org.junit.runners.BlockJUnit4ClassRunner.createTest$original$16gso2gv$accessor$nt1WBOeA(BlockJUnit4ClassRunner.java)
	at org.junit.runners.BlockJUnit4ClassRunner$auxiliary$2HEKf0Uj.call(Unknown Source)
	at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
	at com.nordstrom.automation.junit.CreateTest.intercept(CreateTest.java:36)
	at org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java)
	at org.junit.experimental.theories.Theories$TheoryAnchor$1.createTest$accessor$jNPCaVGY(Theories.java)
	at org.junit.experimental.theories.Theories$TheoryAnchor$1$auxiliary$OKIv9xhR.call(Unknown Source)
	at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
	at com.nordstrom.automation.junit.CreateTest.intercept(CreateTest.java:36)
	at org.junit.experimental.theories.Theories$TheoryAnchor$1.createTest(Theories.java)
	at org.junit.runners.BlockJUnit4ClassRunner$2.runReflectiveCall$original$qdrjh9Tv(BlockJUnit4ClassRunner.java:309)
	at org.junit.runners.BlockJUnit4ClassRunner$2.runReflectiveCall$original$qdrjh9Tv$accessor$3FlV2mmC(BlockJUnit4ClassRunner.java)
	at org.junit.runners.BlockJUnit4ClassRunner$2$auxiliary$xXc0fpDh.call(Unknown Source)
	at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
	at com.nordstrom.automation.junit.RunReflectiveCall.intercept(RunReflectiveCall.java:85)
	at org.junit.runners.BlockJUnit4ClassRunner$2.runReflectiveCall(BlockJUnit4ClassRunner.java)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:306)
	at org.junit.experimental.theories.Theories$TheoryAnchor$1.methodBlock(Theories.java:232)
	at org.junit.experimental.theories.Theories$TheoryAnchor.runWithCompleteAssignment(Theories.java:223)
	at org.junit.experimental.theories.Theories$TheoryAnchor.runWithAssignment(Theories.java:209)
	at org.junit.experimental.theories.Theories$TheoryAnchor.evaluate(Theories.java:192)
	at com.nordstrom.automation.junit.RetryHandler.runChildWithRetry(RetryHandler.java:54)
	at com.nordstrom.automation.junit.RunChild.intercept(RunChild.java:59)
	at org.junit.experimental.theories.Theories.runChild(Theories.java)
	at org.junit.experimental.theories.Theories.runChild(Theories.java)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run$original$KDZK6cAR(ParentRunner.java:413)
	at org.junit.runners.ParentRunner.run$original$KDZK6cAR$accessor$7H1m3s9a(ParentRunner.java)
	at org.junit.runners.ParentRunner$auxiliary$solXL5h8.call(Unknown Source)
	at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
	at com.nordstrom.automation.junit.Run.intercept(Run.java:55)
	at org.junit.runners.ParentRunner.run(ParentRunner.java)
	at org.junit.runners.BlockJUnit4ClassRunner.run$accessor$nt1WBOeA(BlockJUnit4ClassRunner.java)
	at org.junit.runners.BlockJUnit4ClassRunner$auxiliary$D8nLJ3YO.call(Unknown Source)
	at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
	at com.nordstrom.automation.junit.Run.intercept(Run.java:55)
	at org.junit.runners.BlockJUnit4ClassRunner.run(BlockJUnit4ClassRunner.java)
	at org.junit.experimental.theories.Theories.run$accessor$N0MIaorS(Theories.java)
	at org.junit.experimental.theories.Theories$auxiliary$I820y2np.call(Unknown Source)
	at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
	at com.nordstrom.automation.junit.Run.intercept(Run.java:55)
	at org.junit.experimental.theories.Theories.run(Theories.java)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)

I've prepared a pack of tests to reproduce the issue, they are located here: https://github.com/reportportal/agent-java-junit/tree/develop/src/test/java/com/epam/reportportal/junit/features/theory

Here is an example of a simple test which fails:

import org.junit.Assume;
import org.junit.experimental.theories.Theories;
import org.junit.experimental.theories.Theory;
import org.junit.runner.RunWith;

@RunWith(Theories.class)
public class TheoryPassedTest {

	@Theory
	public void theories() {
		Assume.assumeTrue(true);
	}
}

The native implementation of PowerMockRunner uses a deprecated JUnit runner model that JUnit Foundation doesn't support. You need to delegate test execution to the standard BlockJUnit4ClassRunner (or subclasses thereof) to enable reporting of test lifecycle events. This is specified via the @PowerMockRunnerDelegate annotation, as shown below:

package com.nordstrom.automation.junit;

import static org.hamcrest.CoreMatchers.equalTo;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.mockito.Mockito.when;
import static org.powermock.api.mockito.PowerMockito.mockStatic;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;
import org.powermock.modules.junit4.PowerMockRunnerDelegate;

@RunWith(PowerMockRunner.class)
@PowerMockRunnerDelegate(BlockJUnit4ClassRunner.class)
@PrepareForTest(PowerMockCases.StaticClass.class)
public class PowerMockCases {
    
    @Test
    public void testHappyPath() {
        mockStatic(StaticClass.class);
        when(StaticClass.staticMethod()).thenReturn("mocked");
        assertThat(StaticClass.staticMethod(), equalTo("mocked"));
    }
    
    static class StaticClass {
        public static String staticMethod() {
            return null;
        }
    }
}

I've tried Gradle Configuration for JUnit Foundation in my android project and got such build issue

Cannot change resolution strategy of dependency configuration ':app:compile' after it has been resolved.

Steps to reproduce

• Android gradle plugin com.android.tools.build:gradle:4.0.1
• Gradle version 6.1.1
• Create an android project and do setup JUnit Foundation for Gradle Configuration.

• I need to clarify is the difference between getCallableOf and encloseCallable. The first is used to retrieve the callable object created by the runner; the second creates a new object. For most runners, these objects are interchangeable, but for JUnitParamsRunner you don't get the parameters in the new object.
• I need to highlight the isRetriedTest method for its handling of both test methods and theory methods.
• I need to add notes of caution to watcher methods that get called when methods fail. For suite-level failures, the "atomic test" object included with the notification represents the test class, not a specific method.

Some JUnit's test rules may produce different exceptions. Unfortunately JUnit-Foundation library can't properly handle that exceptions and just throws NullPointerException.

While using JUnit-Foundation library with ReportPortal that's mean that a launch won't be finished at all, when this exception appears.

For example, for the following rule:

public class FailableRule implements TestRule {

    @Override
    public Statement apply(Statement statement, Description description) {
        throw new RuntimeException("Must be failed");
    }
}

The following NPE exception will be generated while run test via maven-surefire-plugin:

org.apache.maven.surefire.testset.TestSetFailedException: Test mechanism :: null
at org.apache.maven.surefire.common.junit4.JUnit4RunListener.rethrowAnyTestMechanismFailures(JUnit4RunListener.java:223)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:167)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417)
Caused by: java.lang.NullPointerException
at com.nordstrom.automation.junit.RunAnnouncer.setTestFailure(RunAnnouncer.java:118)
at com.nordstrom.automation.junit.RunAnnouncer.testFailure(RunAnnouncer.java:53)
at org.junit.runner.notification.SynchronizedRunListener.testFailure(SynchronizedRunListener.java:63)
at org.junit.runner.notification.RunNotifier$4.notifyListener(RunNotifier.java:142)
at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72)
at org.junit.runner.notification.RunNotifier.fireTestFailures(RunNotifier.java:138)
at org.junit.runner.notification.RunNotifier.fireTestFailure(RunNotifier.java:132)
at org.apache.maven.surefire.common.junit4.Notifier.fireTestFailure(Notifier.java:114)
at org.junit.internal.runners.model.EachTestNotifier.addFailure(EachTestNotifier.java:23)
at org.junit.runners.ParentRunner.run$original$uXqF4S3L(ParentRunner.java:369)
at org.junit.runners.ParentRunner.run$original$uXqF4S3L$accessor$0KImNepD(ParentRunner.java)
at org.junit.runners.ParentRunner$auxiliary$XI5ofNnD.call(Unknown Source)
at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:359)
at com.nordstrom.automation.junit.Run.intercept(Run.java:57)
at org.junit.runners.ParentRunner.run(ParentRunner.java)
at org.junit.runners.BlockJUnit4ClassRunner.run$accessor$VlVuHR5I(BlockJUnit4ClassRunner.java)
at org.junit.runners.BlockJUnit4ClassRunner$auxiliary$tH4bB6FG.call(Unknown Source)
at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:359)
at com.nordstrom.automation.junit.Run.intercept(Run.java:57)
at org.junit.runners.BlockJUnit4ClassRunner.run(BlockJUnit4ClassRunner.java)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)

Scenario:
Run a test with a Before class method expected to fail

Result:
java.lang.NullPointerException at com.nordstrom.automation.junit.RunAnnouncer.setTestFailure(RunAnnouncer.java:136) at com.nordstrom.automation.junit.RunAnnouncer.testFailure(RunAnnouncer.java:64) at org.junit.runner.notification.SynchronizedRunListener.testFailure(SynchronizedRunListener.java:63) at org.junit.runner.notification.RunNotifier$4.notifyListener(RunNotifier.java:142) at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) at org.junit.runner.notification.RunNotifier.fireTestFailures(RunNotifier.java:138) at org.junit.runner.notification.RunNotifier.fireTestFailure(RunNotifier.java:132) at org.junit.internal.runners.model.EachTestNotifier.addFailure(EachTestNotifier.java:23) at org.junit.runners.ParentRunner.run$original$3KEFEFXu(ParentRunner.java:369) at org.junit.runners.ParentRunner.run$original$3KEFEFXu$accessor$LACdlSbF(ParentRunner.java) at org.junit.runners.ParentRunner$auxiliary$YwrRWEJd.call(Unknown Source) at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:338) at com.nordstrom.automation.junit.Run.intercept(Run.java:66) at org.junit.runners.ParentRunner.run(ParentRunner.java) at org.junit.runners.BlockJUnit4ClassRunner.run$accessor$MQdh69pA(BlockJUnit4ClassRunner.java) at org.junit.runners.BlockJUnit4ClassRunner$auxiliary$IptvJh1v.call(Unknown Source) at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:338) at com.nordstrom.automation.junit.Run.intercept(Run.java:66) at org.junit.runners.BlockJUnit4ClassRunner.run(BlockJUnit4ClassRunner.java) at Utils.junit.Junit4CustomClassRunner.run$accessor$RRbPQNfc(Junit4CustomClassRunner.java) at Utils.junit.Junit4CustomClassRunner$auxiliary$F3wSCyuO.call(Unknown Source) at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:338) at com.nordstrom.automation.junit.Run.intercept(Run.java:66) at Utils.junit.Junit4CustomClassRunner.run(Junit4CustomClassRunner.java) at Utils.junit.TestRunner.run$original$ocyOZSnB(TestRunner.java:63) at Utils.junit.TestRunner.run$original$ocyOZSnB$accessor$TphGJu42(TestRunner.java) at Utils.junit.TestRunner$auxiliary$535EKzJh.call(Unknown Source) at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:338) at com.nordstrom.automation.junit.Run.intercept(Run.java:66) at Utils.junit.TestRunner.run(TestRunner.java) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

Please note that issue "Change tracking of "atomic" test to handle rule failures" handled a similar issue in exception thrown from a rule.
The issue still reproduces in entire Suite failures, because the Map "RUNNER_TO_ATOMICTEST" is populated only from start of a test.

Can you please aid with a solution?
Thanks!

In case tests are being executed with non-standard junit runner, the following class cast exception occurs:

java.lang.ClassCastException: class cucumber.runtime.junit.PickleRunners$NoStepDescriptions cannot be cast to class org.junit.runners.model.FrameworkMethod (cucumber.runtime.junit.PickleRunners$NoStepDescriptions and org.junit.runners.model.FrameworkMethod are in unnamed module of loader 'app')

	at com.nordstrom.automation.junit.AtomicTest.<init>(AtomicTest.java:38)
	at com.nordstrom.automation.junit.RunAnnouncer.newAtomicTest(RunAnnouncer.java:95)
	at com.nordstrom.automation.junit.RunAnnouncer.testStarted(RunAnnouncer.java:27)

Since org.junit.runners.ParentRunner does not explicitly specify object type of children, junit-found seems to be also should not rely on FrameworkMethod only.
For example, for Cucumber JUnit runner children have cucumber.runtime.junit.PickleRunners$NoStepDescriptions type

JUnit Foundation only has a few settings. One of these settings is RULE_CHAIN_LIST. At this point, there's no documentation for this option save the barest hints provided by the option name and its default value.

The code that uses the value of this option provides the remaining insights. The default value for RULE_CHAIN_LIST ("rulesStartingWithInnerMost") is the name of a private field of the RuleChain class, which is needed to acquire the list of rules attached to the chain maintained by each rule chain instance.

The theory behind providing this option is that the name of the field could change at some point, requiring a mechanism to specify the new field name to restore the ability to acquire the rules list. This possibility is remote at best, and I suspect that no one actually uses the RuleChainWalker anyway. I think the best answer to this option is to remove it.

Hello,
When someone uses a PowerMock runner in tests JUnit Foundation does not fire any events in listeners.

E.G.:

import org.apache.commons.lang3.RandomStringUtils;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.equalTo;
import static org.mockito.Mockito.when;
import static org.powermock.api.mockito.PowerMockito.mockStatic;

@RunWith(PowerMockRunner.class)
@PrepareForTest(StaticClass.class)
public class SimplePowermockTest {
	@Test
	public void testPowerMock() {
		String rndStr = RandomStringUtils.randomAlphanumeric(10);
		mockStatic(StaticClass.class);
		when(StaticClass.staticMethod()).thenReturn(rndStr);
		assertThat(StaticClass.staticMethod(), equalTo(rndStr));
	}
}

I've prepared two Test Cases to reproduce the issue:

• https://github.com/reportportal/agent-java-junit/blob/develop/src/test/java/com/epam/reportportal/junit/features/powermock/SimplePowermockTest.java
• https://github.com/reportportal/agent-java-junit/blob/develop/src/test/java/com/epam/reportportal/junit/features/powermock/SimplePowermockFailureTest.java

And here are corresponding tests, which catch the issue (currently disabled):

• https://github.com/reportportal/agent-java-junit/blob/develop/src/test/java/com/epam/reportportal/junit/powermock/PowerMockPositiveTest.java
• https://github.com/reportportal/agent-java-junit/blob/develop/src/test/java/com/epam/reportportal/junit/powermock/PowerMockNegativeTest.java

The bare-bones JavaDoc that's currently attached to constants in the JUnitSettings enumeration is wholly inadequate. Follow the model set by Selenium Foundation and expand this documentation so that it's actually informative and useful.

Hello.
I'm trying to use JUnit-Foundation on an Android gradle project.
I follow your setup and added the

testOptions {
        unitTests.all {
            jvmArgs "-javaagent:${classpath.find { it.name.contains('junit-foundation') }.absolutePath}"
        }
    }

section but when I try to synchronize gradle (or run a build) I get alway the
Cannot get property 'absolutePath' on null object
error.
Any advice?
Here my (partial) build.gradle

android {
    compileSdkVersion 28

    compileOptions {
        incremental true
        // Sets Java compatibility to Java 8
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }


    defaultConfig {
        applicationId "com.inpeco.protube.mobile"
        minSdkVersion 28
        targetSdkVersion 28
        versionCode 1
        versionName "2.2.0"
    }
}    
dependencies {
    testImplementation 'net.bytebuddy:byte-buddy:1.10.14' // I should add this due to classNotFoundExceptions
    testImplementation 'com.nordstrom.tools:junit-foundation:13.0.0'
    
    ...
}

Hi there,

First of all, nice work with this tool.

I tried running it against some Java 11 compiled code (using gradle) but encountered some issues:

Starting process 'Gradle Test Executor 3507'. Command: /usr/java/jdk11/bin/java -Djunit.max.retry=2 -Djunit.timeout.test=500 -Dorg.gradle.native=false -javaagent:/home/ovidiu/.gradle/caches/modules-2/files-2.1/com.nordstrom.tools/junit-foundation/6.1.4/1f6ea951c06fb8a19ba0e59408a4ca7a82e34d50/junit-foundation-6.1.4.jar -javaagent:/home/ovidiu/.gradle/caches/modules-2/files-2.1/org.jmockit/jmockit/1.28/3efdc615563d9fe2ebb29690cae1f33d1187cee6/jmockit-1.28.jar @/tmp/gradle-worker-classpath7097349747053025687txt -Xms256m -Xmx1g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 3507'
Successfully started process 'Gradle Test Executor 3506'
Successfully started process 'Gradle Test Executor 3507'
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
Caused by: java.lang.NoSuchMethodError: net.bytebuddy.pool.TypePool$Default.ofClassPath()Lnet/bytebuddy/pool/TypePool;
        at com.nordstrom.automation.junit.LifecycleHooks.installTransformer(LifecycleHooks.java:74)
        at com.nordstrom.automation.junit.LifecycleHooks.premain(LifecycleHooks.java:64)
        ... 6 more
FATAL ERROR in native method: processing of -javaagent failed


$ less /tmp/gradle-worker-classpath7097349747053025687txt | grep -e byte -e foundation
/home/ovidiu/.gradle/caches/modules-2/files-2.1/com.nordstrom.tools/junit-foundation/6.1.4/1f6ea951c06fb8a19ba0e59408a4ca7a82e34d50/junit-foundation-6.1.4.jar
/home/ovidiu/.gradle/caches/modules-2/files-2.1/net.bytebuddy/byte-buddy/1.9.3/f32e510b239620852fc9a2387fac41fd053d6a4d/byte-buddy-1.9.3.jar
/home/ovidiu/.gradle/caches/modules-2/files-2.1/net.bytebuddy/byte-buddy-agent/1.9.3/f5b78c16cf4060664d80b6ca32d80dca4bd3d264/byte-buddy-agent-1.9.3.jar

• Gradle: 5.0-rc-1
• bytebuddy : 1.9.3
  • Java 11 support was added from 1.9.x
• junit-foundation: 6.1.4

From a quick look over the bytebuddy changes I found this one: raphw/byte-buddy@8cc6f79#diff-6e2e25e8562b392b2b4f72c4648289bd

Tried a quick fix locally (I can do a PR if needed as well)
upgrade_bytebuddy.txt

Regards,
Ovidiu

These issues were reported against junit-foundation-6.3.1:

• When running in parallel, event notifications got sent with the wrong context.
• In many scenarios with parameterized tests, the event notification code would throw exceptions (e.g. - NullPointerException), resulting in false test failures.
• Due to missing intercept hooks, we would miss key events.
• There was no code in place to recognize nested interceptions, resulting in duplicate notifications.

When tests are ran with RunAnnouncer listener suite/test run is not intercepted when tests are marked with @ignore on class level.
If test are marked with @ignore on method level it works fine.

Looks like suite/test start is not intercepted probably because another Junit runner is used: IgnoredClassRunner. Thus AtomicTest information is not gathered. Later when testIgnored method is invoked it tries to get test identity on null object and fails with null.
This happens in:
private static boolean isSupported(RunWatcher<?> watcher, AtomicTest<?> atomicTest) { return watcher.supportedType().isInstance(atomicTest.getIdentity()); }

Can be tested from branch https://github.com/reportportal/examples-java/tree/bug-npe-for-ignored-class
Test class:
https://github.com/reportportal/examples-java/blob/bug-npe-for-ignored-class/example-junit/src/test/java/com/epam/reportportal/example/junit/extension/RetryTest.java

Please note that RunAnnounce listener should be added to maven surefire plugin

The README file is quite long, and it takes a lot of dedication to acquire an understanding of the JUnit Foundation value proposition. Follow the pattern established by Selenium Foundation, adding a TOC and Highlight section to introduce features and guide readers to the content that interests them.

I had to make significant revisions to resolve memory consumption issues caused by unmanaged retention of framework object references. To ensure that I don't introduce regressions in future revisions, I need to add unit tests to verify that all collected references are released at key points of execution for all supported scenarios:

• Standard happy-path execution
• Pass on automatic retry
• Fail on automatic retry
• Method-specific timeout
• Rule-based timeout
• Parameterized runner
• JUnitParams runner
• Theories runner
• PowerMock runner

Checks for reference retention must be performed at relevant points in the flow of execution. This can probably be accomplished with a properly crafted watcher/listener class.