testingresearchillinois / testrunner Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 10.0 167 KB

Java 68.79% Scala 31.21%

testrunner's People

Contributors

Stargazers

Watchers

Forkers

luoos byjiang1996 wenxiwang unshorn lcpchopin lalayang bedomagdi lcpchopin

testrunner's Issues

Fail to Exclude Disabled Test Classes and Methods

During the finding phase for JUnit 5 tests, Disabled classes and methods should be excluded. But they are not excluded. They can still be found in the "original-order" file.

Example:

Disabled class: CqlLibraryConverterTest
Disabled test: disabledTest

Fail to Run JUnit 5 Tests with Parameter

The current JUnit 5 executor fails to run JUnit 5 tests with parameters. For example, DefaultCassandraDatabaseFactoryTests. configureJavaHomeEnvVariable.

Throw this exception:

ERROR (WRITE_ERROR_STDOUT_THROWABLE): com.github.nosan:embedded-cassandra:4.0.0-SNAPSHOT
java.lang.RuntimeException: Set of executed tests is not equal to test list that should have been executed (397 tests executed, 441 tests expected). Missing tests are: Set(com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setConfigPropertyUri, com.github.nosan.embedded.cassandra.WorkingDirectoryCustomizerTests#shouldNotCopyOutOfDirectory, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setConfigFileUrl, com.github.nosan.embedded.cassandra.commons.FileLockTests#zeroTimeout, com.github.nosan.embedded.cassandra.commons.FileLockTests#successTryLock, com.github.nosan.embedded.cassandra.WebCassandraDirectoryProviderTests#directoryPresentJustReturn, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setConfigPropertyUrl, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#configureSeedsConfigStoragePort, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#configureEnvVariables, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#configureSeeds, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setConfigPropertiesReplaceRandomPorts, com.github.nosan.embedded.cassandra.DefaultSettingsTests#checkConstruct, com.github.nosan.embedded.cassandra.WorkingDirectoryCustomizerTests#copy, com.github.nosan.embedded.cassandra.DefaultCassandraIntegrationTests#getWorkingDirectory, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setConfigPropertyUriResource, com.github.nosan.embedded.cassandra.commons.FileSystemResourceTests#isWritable, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setConfigPropertyInetAddress, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#configureSeedsSystemStoragePort, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setConfigPropertyClassPathResource, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#configSystemProperties, com.github.nosan.embedded.cassandra.WorkingDirectoryDestroyerTests#deleteAll, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setConfigPropertiesCannotHaveNestedProperties, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setSystemPropertiesReplaceRandomPorts, com.github.nosan.embedded.cassandra.WorkingDirectoryDestroyerTests#doNothing, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setConfigPropertyPath, com.github.nosan.embedded.cassandra.DefaultWorkingDirectoryInitializerTests#initializeWorkingDir, com.github.nosan.embedded.cassandra.WorkingDirectoryDestroyerTests#deleteOnly, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#configureJavaHomeEnvVariable, com.github.nosan.embedded.cassandra.CassandraBuilderTests#workingDirectory, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#configJvmOptions, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setConfigFileResource, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setConfigPropertyFile, com.github.nosan.embedded.cassandra.WebCassandraDirectoryProviderTests#construct3, com.github.nosan.embedded.cassandra.WorkingDirectoryDestroyerTests#shouldNotDeleteOutOfDirectory, com.github.nosan.embedded.cassandra.commons.FileLockTests#invalidTimeout, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setConfigFilePath, com.github.nosan.embedded.cassandra.WorkingDirectoryCustomizerTests#shouldNotCopyTargetIsDirectory, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setConfigFileFile, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setConfigProperties, com.github.nosan.embedded.cassandra.WebCassandraDirectoryProviderTests#failToMoveDownloadedShouldBeCopy, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setConfigPropertyUrlResource, com.github.nosan.embedded.cassandra.commons.FileSystemResourceTests#isReadable, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setConfigFileString, com.github.nosan.embedded.cassandra.DefaultCassandraDatabaseFactoryTests#setConfigFileUri)
	at edu.illinois.cs.testrunner.runner.SmartRunner.$anonfun$run$1(SmartRunner.scala:40)
	at scala.util.Success.flatMap(Try.scala:247)
	at edu.illinois.cs.testrunner.runner.SmartRunner.run(SmartRunner.scala:33)
	at edu.illinois.cs.testrunner.runner.Runner.runList(Runner.scala:28)
	at edu.illinois.cs.testrunner.runner.Runner.runList$(Runner.scala:27)
	at edu.illinois.cs.testrunner.runner.SmartRunner.runList(SmartRunner.scala:21)
	at edu.illinois.cs.dt.tools.detection.DetectorUtil.originalResults(DetectorUtil.java:32)
	at edu.illinois.cs.dt.tools.detection.detectors.RandomDetector.<init>(RandomDetector.java:26)
	at edu.illinois.cs.dt.tools.detection.detectors.DetectorFactory.makeDetector(DetectorFactory.java:19)
	at edu.illinois.cs.dt.tools.detection.DetectorPlugin.detectorExecute(DetectorPlugin.java:239)
	at edu.illinois.cs.dt.tools.detection.DetectorPlugin.lambda$execute$5(DetectorPlugin.java:201)
	at edu.illinois.cs.dt.tools.utility.ErrorLogger.runAndLogError(ErrorLogger.java:39)
	at edu.illinois.cs.dt.tools.detection.DetectorPlugin.execute(DetectorPlugin.java:201)

Failed to run idflakies on following projects

The latest testrunner(1.1)/iDFlakies(1.1.0) failed to run iDFlakies on Sentinel repo:

Project: https://github.com/alibaba/Sentinel
Branch: master
SHA: 8c4f251922365029a36ca6fe912b8aa88d6fe286
Module/submodule name: sentinel-demo-quarkus

Error logs:

ERROR (WRITE_ERROR_STDOUT_THROWABLE): com.alibaba.csp:sentinel-demo-quarkus:1.8.1-SNAPSHOT
java.lang.RuntimeException: Set of executed tests is not equal to test list that should have been executed (2 tests executed, 4 tests expected). Did you use testplugin.runner.idempotent.num.runs? Missing tests are: Set(com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelAnnotationQuarkusAdapter(), com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelJaxRsQuarkusAdapter())
at edu.illinois.cs.testrunner.runner.SmartRunner.$anonfun$run$1(SmartRunner.scala:44)
at scala.util.Success.flatMap(Try.scala:247)
at edu.illinois.cs.testrunner.runner.SmartRunner.run(SmartRunner.scala:36)
at edu.illinois.cs.testrunner.runner.Runner.runList(Runner.scala:28)
at edu.illinois.cs.testrunner.runner.Runner.runList$(Runner.scala:27)
at edu.illinois.cs.testrunner.runner.SmartRunner.runList(SmartRunner.scala:22)
at edu.illinois.cs.dt.tools.detection.DetectorUtil.originalResults(DetectorUtil.java:32)
at edu.illinois.cs.dt.tools.detection.detectors.RandomDetector.(RandomDetector.java:26)
at edu.illinois.cs.dt.tools.detection.detectors.DetectorFactory.makeDetector(DetectorFactory.java:19)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.detectorExecute(DetectorPlugin.java:240)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.lambda$execute$5(DetectorPlugin.java:201)
at edu.illinois.cs.dt.tools.utility.ErrorLogger.runAndLogError(ErrorLogger.java:39)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.execute(DetectorPlugin.java:201)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at edu.illinois.cs.testrunner.mavenplugin.TestPluginPlugin.execute(TestPluginPlugin.scala:102)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)

Lack of support for JUnit 3 tests

testrunner appears unable to control the order of methods within a class for JUnit 3 tests.

The order of test classes for JUnit 3 tests is able to be controlled but the method order within a class is not controlled.

The same problem is in our custom Maven surefire plugin.
https://github.com/TestingResearchIllinois/maven-surefire/tree/test-method-sorting

To reproduce create two simple tests that depend on a variable x and assert different values of x with one of the test changing x. Adding println to see the order of the tests would also help.

Running something like the following will always see the tests pass or fail as the order is always the same.
rm -rf .dtfixingtools/; mkdir .dtfixingtools/; cp original-order .dtfixingtools/original-order; mvn testrunner:testplugin -Ddetector.detector_type=random-class-method -Ddt.randomize.rounds=5 -Ddt.detector.original_order.all_must_pass=false

OutOfMemoryError

When running iDFlakies on a project with a lot of tests, testrunner sometimes (not 100%) throws OutOfMemoryError.

Project: gradle/performance-comparisons
Branch: commons-lang (there is only one commit)
Version: testrunner-maven-plugin 1.0; idflakies 1.0.2 (I believe this issue also exists in the latest version 1.1)
Command: mvn testrunner:testplugin -Ddetector.detector_type=random-class-method -Ddt.randomize.rounds=10 -Ddt.detector.original_order.all_must_pass=false
Number of tests: 3120
Note: There are supposed to be 10 rounds, the error was thrown at the tenth round

ERROR (WRITE_ERROR_STDOUT_THROWABLE): org.apache.commons:commons-lang3:3.6-SNAPSHOT
ava.lang.OutOfMemoryError: Java heap space
	at java.util.ArrayList.<init>(ArrayList.java:183)
	at edu.illinois.cs.testrunner.runner.TestInfo.beforeInc(TestInfo.java:16)
	at edu.illinois.cs.testrunner.runner.TestInfo.updateFlakiness(TestInfo.java:50)
	at edu.illinois.cs.testrunner.runner.TestInfo.updateWith(TestInfo.java:39)
	at edu.illinois.cs.testrunner.runner.TestInfoStore.lambda$update$1(TestInfoStore.java:46)
	at edu.illinois.cs.testrunner.runner.TestInfoStore$$Lambda$77/531462687.accept(Unknown Source)
	at java.util.Map.forEach(Map.java:630)
	at edu.illinois.cs.testrunner.runner.TestInfoStore.update(TestInfoStore.java:44)
	at edu.illinois.cs.testrunner.runner.TestInfoStore.lambda$update$0(TestInfoStore.java:34)
	at edu.illinois.cs.testrunner.runner.TestInfoStore$$Lambda$76/1424610562.apply(Unknown Source)
	at scala.Option.foreach(Option.scala:257)
	at edu.illinois.cs.testrunner.runner.TestInfoStore.update(TestInfoStore.java:33)
	at edu.illinois.cs.testrunner.runner.SmartRunner.run(SmartRunner.scala:30)
	at edu.illinois.cs.testrunner.runner.Runner.runList(Runner.scala:28)
	at edu.illinois.cs.testrunner.runner.Runner.runList$(Runner.scala:27)
	at edu.illinois.cs.testrunner.runner.SmartRunner.runList(SmartRunner.scala:21)
	at edu.illinois.cs.dt.tools.detection.detectors.ExecutingDetector.runList(ExecutingDetector.java:50)
	at edu.illinois.cs.dt.tools.detection.detectors.RandomDetector.results(RandomDetector.java:42)
	at edu.illinois.cs.dt.tools.detection.detectors.ExecutingDetector$RunnerIterator.generateDetectionRound(ExecutingDetector.java:133)
	at edu.illinois.cs.dt.tools.detection.detectors.ExecutingDetector$RunnerIterator.generate(ExecutingDetector.java:150)

Fail to Apply iDFlakies on apache/metron (Junit5)

apache/metron uses JUnit 5. iDFlakies/Testrunner fail to run most of its tests (some tests are successful).

Sample exceptions:

ERROR (WRITE_ERROR_STDOUT_THROWABLE): org.apache.metron:metron-parsers:0.7.2
java.lang.RuntimeException: Set of executed tests is not equal to test list that should have been executed (0 tests executed, 131 tests expected). Did you use testplugin.runner.idempotent.num.runs? Missing tests are: Set(org.apache.metron.parsers.bro.BasicBroParserTest#testX509BroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testConnBroMessage(), org.apache.metron.parsers.leef.LEEFParserTest#testFourDayFutureBecomesPast(), org.apache.metron.parsers.bro.BasicBroParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.cef.CEFParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.SnortParserTest#testGoodMessage(), org.apache.metron.parsers.cef.CEFParserTest#testTimestampPriority(), org.apache.metron.parsers.SnortParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEFdevTimeWithNoCustomFormat(), org.apache.metron.parsers.lancope.BasicLancopeParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.YafParserTest#test(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEF_2_0(), org.apache.metron.parsers.asa.BasicAsaParserTest#testCISCOFW106023(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.fireeye.BasicFireEyeParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.fireeye.BasicFireEyeParserTest#testParse(), org.apache.metron.parsers.asa.BasicAsaParserTest#testConfigureDefault(), org.apache.metron.parsers.asa.BasicAsaParserTest#testSyslogHostAndProg(), org.apache.metron.parsers.cef.CEFParserTest#testMissingYearFromDate(), org.apache.metron.parsers.sourcefire.BasicSourcefireParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.bro.BasicBroParserTest#testWeirdBroMessage(), org.apache.metron.parsers.asa.BasicAsaParserTest#testNoPatternForTag(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseInvalidLogTypeMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testSoftwareBroMessage(), org.apache.metron.parsers.SnortParserTest#throws_exception_on_bad_config_date_format(), org.apache.metron.parsers.bro.BasicBroParserTest#testDhcpBroMessage(), org.apache.metron.filters.FiltersTest#testSingleQueryFilter(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEFcustomdevTimeFormat(), org.apache.metron.parsers.bro.BasicBroParserTest#testKnownDevicesBroMessage(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#testParseMalformedOtherLine(), org.apache.metron.parsers.cef.CEFParserTest#testCEFParserCyberArk(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseConfig61CustomFields(), org.apache.metron.parsers.ise.BasicIseParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.leef.LEEFParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEFMultiLine(), org.apache.metron.parsers.asa.BasicAsaParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseTraffic71(), org.apache.metron.parsers.asa.BasicAsaParserTest#testIp6Addr(), org.apache.metron.parsers.bro.BasicBroParserTest#testHttpBroMessage(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseThreat80(), org.apache.metron.parsers.asa.BasicAsaParserTest#testCISCOFW106006(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#testParseLogoutLine(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseThreat71(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseConfig70And80NoCustomFields(), org.apache.metron.parsers.bro.BasicBroParserTest#testBadMessage(), org.apache.metron.parsers.leef.LEEFParserTest#testMissingYearFromDate(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#testParseRBMLine(), org.apache.metron.parsers.cef.CEFParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.asa.BasicAsaParserTest#testUnexpectedMessage(), org.apache.metron.parsers.fireeye.BasicFireEyeParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.bro.BasicBroParserTest#testRadiusBroMessageSuccess(), org.apache.metron.parsers.cef.CEFParserTest#testCEFParserAdallom(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseTraffic60(), org.apache.metron.parsers.lancope.BasicLancopeParserTest#testParse(), org.apache.metron.parsers.leef.LEEFParserTest#testTimestampPriority(), org.apache.metron.parsers.asa.BasicAsaParserTest#testConfigureTimeZoneOffset(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#testParseMalformedLogoutLine(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEF_1_0_versionIncluded(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEF_2_0_delimiterSpecified(), org.apache.metron.parsers.bro.BasicBroParserTest#testFtpBroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testUnwrappedBroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testDpdBroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testRfbBroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testNoticeBroMessage(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#testParseMalformedRBMLine(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseConfig70And80CustomFields(), org.apache.metron.parsers.sourcefire.BasicSourcefireParserTest#testParse(), org.apache.metron.parsers.cef.CEFParserTest#testPaloAltoCEF(), org.apache.metron.parsers.bro.BasicBroParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.bro.BasicBroParserTest#testHttpBroMessageWithZeroDecimalTruncation(), org.apache.metron.parsers.SnortParserTest#throws_exception_on_bad_config_timezone(), org.apache.metron.parsers.bro.BasicBroParserTest#testSoftwareBroMessage2(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseTraffic80(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseConfig61NoCustomFields(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseTraffic70(), org.apache.metron.parsers.bro.BasicBroParserTest#testRadiusBroMessageFailed(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.bro.BasicBroParserTest#testFilesBroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testDnsBroMessage(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#testParseOtherLine(), org.apache.metron.parsers.bro.BasicBroParserTest#testDecimalFormatAssumptions(), org.apache.metron.parsers.bro.BasicBroParserTest#testSmtpBroMessage(), org.apache.metron.parsers.asa.BasicAsaParserTest#testSyslogIpHost(), org.apache.metron.parsers.lancope.BasicLancopeParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.bro.BasicBroParserTest#testHttpBroDecimalMessage(), org.apache.metron.parsers.asa.BasicAsaParserTest#testConfigureTimeZoneText(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseThreat70(), org.apache.metron.parsers.bro.BasicBroParserTest#testSshBroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testSslBroMessage(), org.apache.metron.parsers.asa.BasicAsaParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#testParseMalformedLoginLine(), org.apache.metron.parsers.asa.BasicAsaParserTest#testShortTimestamp(), org.apache.metron.parsers.sourcefire.BasicSourcefireParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.SnortParserTest#parses_timestamp_as_local_zone_by_default(), org.apache.metron.parsers.bro.BasicBroParserTest#testCaptureLossBroMessage(), org.apache.metron.parsers.leef.LEEFParserTest#testInvalid(), org.apache.metron.parsers.bro.BasicBroParserTest#testStatsBroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testSipBroMessage(), org.apache.metron.parsers.cef.CEFParserTest#testEscaping(), org.apache.metron.parsers.cef.CEFParserTest#testFourDayFutureBecomesPast(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEF_2_0_delimiterUsedIncorrectly(), org.apache.metron.parsers.cef.CEFParserTest#testBasicHeader(), org.apache.metron.parsers.cef.CEFParserTest#testCustomLabelWithSpace(), org.apache.metron.parsers.SnortParserTest#testBadMessage(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#testParseLoginLine(), org.apache.metron.parsers.cef.CEFParserTest#testRtValueAsEpochTimestamp(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseSystem61(), org.apache.metron.parsers.asa.BasicAsaParserTest#testSyslogHost(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.cef.CEFParserTest#testCEFParserWAF(), org.apache.metron.parsers.bro.BasicBroParserTest#testProtocolKeyCleanedUp(), org.apache.metron.parsers.SnortParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseThreat60(), org.apache.metron.parsers.ise.BasicIseParserTest#testParse(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEFParserSample(), org.apache.metron.parsers.leef.LEEFParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.asa.BasicAsaParserTest#testInvalidIpAddr(), org.apache.metron.parsers.ise.BasicIseParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseInvalidVersionMessage(), org.apache.metron.parsers.SnortParserTest#uses_configuration_to_parse(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseSystem80(), org.apache.metron.parsers.cef.CEFParserTest#testBasicExtensions(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEF_CEFlikeSample(), org.apache.metron.parsers.cef.CEFParserTest#testInvalid(), org.apache.metron.parsers.SquidParserTest#test(), org.apache.metron.parsers.fireeye.BasicFireEyeParserTest#testTimestampParsing(), org.apache.metron.filters.FiltersTest#testDefault(), org.apache.metron.parsers.cef.CEFParserTest#testSuccessfulWhenCEFContainsJSON(), org.apache.metron.parsers.bro.BasicBroParserTest#testKnownCertsBroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testBadMessageNonJson())
	at edu.illinois.cs.testrunner.runner.SmartRunner.$anonfun$run$1(SmartRunner.scala:44)
	at scala.util.Success.flatMap(Try.scala:247)
	at edu.illinois.cs.testrunner.runner.SmartRunner.run(SmartRunner.scala:36)
	at edu.illinois.cs.testrunner.runner.Runner.runList(Runner.scala:28)
	at edu.illinois.cs.testrunner.runner.Runner.runList$(Runner.scala:27)
	at edu.illinois.cs.testrunner.runner.SmartRunner.runList(SmartRunner.scala:22)
	at edu.illinois.cs.dt.tools.detection.DetectorUtil.originalResults(DetectorUtil.java:32)
	at edu.illinois.cs.dt.tools.detection.detectors.RandomDetector.<init>(RandomDetector.java:26)

ERROR (WRITE_ERROR_STDOUT_THROWABLE): org.apache.metron:metron-pcap-backend:0.7.2
java.lang.RuntimeException: Set of executed tests is not equal to test list that should have been executed (0 tests executed, 24 tests expected). Did you use testplugin.runner.idempotent.num.runs? Missing tests are: Set(org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#date_range_filters_out_all_results(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#filters_pcaps_by_start_end_ns_with_fixed_filter(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#fixed_filter_returns_all_results_for_full_date_range(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#protocol_filters_out_all_results_with_fixed_filter(), org.apache.metron.pcap.query.PcapCliTest#missing_start_time_arg_prints_error_and_help(), org.apache.metron.pcap.query.PcapCliTest#runs_query_pcap_filter_job_with_default_argument_list(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#filters_pcaps_by_start_end_ns_with_empty_query_filter(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#filters_results_by_dst_port_range_with_query_filter(), org.apache.metron.pcap.query.PcapCliTest#invalid_fixed_filter_arg_prints_help(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#query_filter_returns_all_results_for_full_date_range(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#ip_address_filters_out_all_results_with_fixed_filter(), org.apache.metron.spout.pcap.deserializer.FromKeyDeserializerTest#empty_or_null_key_throws_illegal_argument_exception(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#filters_results_by_BYTEARRAY_MATCHER_with_query_filter(), org.apache.metron.pcap.query.PcapCliTest#invalid_query_filter_arg_prints_help(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#filters_results_by_dst_port_greater_than_value_with_query_filter(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#filters_results_by_dst_port_with_fixed_filter(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#testTimestampInPacket(), org.apache.metron.pcap.query.PcapCliTest#runs_fixed_pcap_filter_job_with_default_argument_list(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#filters_results_by_dst_port_with_query_filter(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#ip_address_filters_out_all_results_with_query_filter(), org.apache.metron.pcap.query.PcapCliTest#runs_fixed_pcap_filter_job_with_full_argument_list(), org.apache.metron.pcap.query.PcapCliTest#runs_fixed_pcap_filter_job_with_full_argument_list_and_default_dateformat(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#protocol_filters_out_all_results_with_query_filter(), org.apache.metron.pcap.query.PcapCliTest#runs_query_pcap_filter_job_with_full_argument_list())
	at edu.illinois.cs.testrunner.runner.SmartRunner.$anonfun$run$1(SmartRunner.scala:44)
	at scala.util.Success.flatMap(Try.scala:247)
	at edu.illinois.cs.testrunner.runner.SmartRunner.run(SmartRunner.scala:36)
	at edu.illinois.cs.testrunner.runner.Runner.runList(Runner.scala:28)
	at edu.illinois.cs.testrunner.runner.Runner.runList$(Runner.scala:27)
	at edu.illinois.cs.testrunner.runner.SmartRunner.runList(SmartRunner.scala:22)
	at edu.illinois.cs.dt.tools.detection.DetectorUtil.originalResults(DetectorUtil.java:32)
	at edu.illinois.cs.dt.tools.detection.detectors.RandomDetector.<init>(RandomDetector.java:26)
	at edu.illinois.cs.dt.tools.detection.detectors.DetectorFactory.makeDetector(DetectorFactory.java:19)

ERROR (WRITE_ERROR_STDOUT_THROWABLE): org.apache.metron:metron-solr-storm:0.7.2
ava.lang.RuntimeException: Set of executed tests is not equal to test list that should have been executed (0 tests executed, 1 tests expected). Did you use testplugin.runner.idempotent.num.runs? Missing tests are: Set(org.apache.metron.indexing.integration.SolrIndexingIntegrationTest#test())
	at edu.illinois.cs.testrunner.runner.SmartRunner.$anonfun$run$1(SmartRunner.scala:44)
	at scala.util.Success.flatMap(Try.scala:247)
	at edu.illinois.cs.testrunner.runner.SmartRunner.run(SmartRunner.scala:36)
	at edu.illinois.cs.testrunner.runner.Runner.runList(Runner.scala:28)
	at edu.illinois.cs.testrunner.runner.Runner.runList$(Runner.scala:27)
	at edu.illinois.cs.testrunner.runner.SmartRunner.runList(SmartRunner.scala:22)
	at edu.illinois.cs.dt.tools.detection.DetectorUtil.originalResults(DetectorUtil.java:32)
	at edu.illinois.cs.dt.tools.detection.detectors.RandomDetector.<init>(RandomDetector.java:26)
	at edu.illinois.cs.dt.tools.detection.detectors.DetectorFactory.makeDetector(DetectorFactory.java:19)

Refactor Needed for "Run Separately" Method (JUnit 5)

To gain full control of test execution order, the current implementation is to run test "separately". There are at least two drawbacks of this method:

@BeforAll/@AfterAll method will execute multiple times. This violates the intention of it, i.e., that method should only run once for all tests in the same test class. And it could lead to some unexpected results.
It brings some performance overhead. But this issue is minor.

Fail to Run JUnit 5 Tests of Nested Class

The JUnit 5 Executor fails to run tests of nested classes. For example, StringDistancesTest.CreateStringDistances#damerauLevenshtein

Exception:

ERROR (WRITE_ERROR_STDOUT_THROWABLE): com.github.mpkorstanje:simmetrics-core:4.1.2-SNAPSHOT
java.lang.RuntimeException: Set of executed tests is not equal to test list that should have been executed (53 tests executed, 192 tests expected). Missing tests are: Set(org.simmetrics.metrics.StringDistancesTest.CreateStringDistances#longestCommonSubsequence, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testSetMetric, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testListMetricWithFilterAndTransform02, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testListDistanceWithFilterAndTransform01, org.simmetrics.metrics.StringMetricsTest.CreateStringMetrics#smithWatermanGotoh, org.simmetrics.builders.StringMetricsTest.CreateForSet#shouldReturnForSetWithSimplifier, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testListDistance02, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testListMetricWithSimplifier01WithCache, org.simmetrics.metrics.StringMetricsTest.CreateStringMetrics#needlemanWunch, org.simmetrics.metrics.HammingDistanceTest.DistanceString#shouldThrowForDifferentLength, org.simmetrics.builders.StringDistancesTest.Create#shouldReturnForSetWithSimplifier, org.simmetrics.metrics.StringMetricsTest.CreateStringMetrics#longestCommonSubsequence, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testListDistanceWithFilterAndTransform02, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testSetMetricWithSimplifier01, org.simmetrics.builders.StringMetricsTest.Create#shouldReturnForListWithSimplifier, org.simmetrics.metrics.StringDistancesTest.CreateStringDistances#hammingDistance, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testSetMetricWithTransform, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testStringMetric01, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testStringDistanceWithSimplifer02WithCache, org.simmetrics.builders.StringDistancesTest.CreateForMultiset#shouldReturnForSet, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testListDistanceWithFilter, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testListMetric01WithCache, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testSetDistanceWithFilterAndTransform01, org.simmetrics.metrics.StringMetricsTest.CreateStringMetrics#smithWaterman, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testStringMetricWithSimplifier02, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testListMetric, org.simmetrics.metrics.DamerauLevenshteinTest.OutofBounds#shouldThrowForDeleteCostOnBound, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testMultisetDistanceWithSimplifier01, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testMultisetDistance, org.simmetrics.builders.StringDistancesTest.CreateForList#shouldReturnForListWithSimplifier, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testMultisetMetricWithSimplifier01WithCache, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testStringMetricWithSimplifier01WithCache, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testListMetric02, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testMultisetDistance02, org.simmetrics.builders.StringDistancesTest.Create#shouldReturnForStringWithChainedSimplifiers, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testMultisetMetric, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testSetDistanceWithSimplifier02WithCache, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testMultisetDistanceWithSimplifier02WithCache, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testStringDistanceWithSimplifier01WithCache, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testMultisetMetric02WithCache, org.simmetrics.builders.StringMetricsTest.Create#shouldReturnForSetWithChainedSimplifiers, org.simmetrics.metrics.StringDistancesTest.CreateStringDistances#longestCommonSubstring, org.simmetrics.builders.StringDistancesTest.Create#shouldReturnForStringWithSimplifier, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testSetMetricWithFilter, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testSetDistanceWithSimplifier02, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testListMetric02WithCache, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testSetMetric02WithCache, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testSetDistance02, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testStringMetricWithSimplifer02WithCache, org.simmetrics.simplifiers.SimplifiersTest.ShouldThrowFor#chainWithNull, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testMultisetMetricWithFilterAndTransform02, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testListMetricWithFilterAndTransform01, org.simmetrics.metrics.StringDistancesTest.Utilities#blockMetric, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testSetDistanceWithTransform, org.simmetrics.builders.StringMetricsTest.CreateForList#shouldReturnForListWithSimplifier, org.simmetrics.builders.StringMetricsTest.Create#shouldReturnSame, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testSetMetricWithFilterAndTransform02, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testSetMetricWithSimplifier02WithCache, org.simmetrics.simplifiers.SimplifiersTest.ShouldThrowFor#chainWithListContainingNull, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testListMetricWithFilter, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testMultisetDistanceWithSimplifier02, org.simmetrics.builders.StringDistancesTest.CreateForList#shouldReturnForList, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testSetDistance02WithCache, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testMultisetMetricWithTransform, org.simmetrics.metrics.DamerauLevenshteinTest.OutofBounds#shouldThrowForDeleteCostBelowBound, org.simmetrics.metrics.StringMetricsTest.CreateStringMetrics#jaro, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testSetDistanceWithFilterAndTransform02, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testMultisetDistanceWithFilterAndTransform01, org.simmetrics.builders.StringDistancesTest.Create#shouldReturnForSetWithChainedSimplifiers, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testListDistanceWithTransform, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testMultisetMetricWithSimplifier02, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testMultisetDistance02WithCache, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testListDistance, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testStringDistanceWithSimplifier01, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testSetDistanceWithSimplifier01WithCache, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testMultisetMetricWithSimplifier01, org.simmetrics.builders.StringMetricsTest.Create#shouldReturnForStringWithSimplifier, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testStringMetricWithSimplifier01, org.simmetrics.builders.StringDistancesTest.Create#shouldReturnSame, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testSetDistanceWithFilter, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testListDistance01WithCache, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testMultisetMetric01WithCache, org.simmetrics.metrics.DamerauLevenshteinTest.OutofBounds#shouldThrowForTransposeCostBelowBound, org.simmetrics.builders.StringMetricsTest.CreateForMultiset#shouldReturnForSetWithSimplifier, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testSetDistanceWithSimplifier01, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testMultisetDistanceWithFilterAndTransform02, org.simmetrics.builders.StringDistancesTest.Create#shouldReturnForListWithChainedSimplifiers, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testSetMetricWithFilterAndTransform01, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testMultisetMetricWithFilterAndTransform01, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testSetDistance01WithCache, org.simmetrics.builders.StringDistancesTest.CreateForSet#shouldReturnForSetWithSimplifier, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testMultisetMetric02, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testListDistanceWithSimplifier02WithCache, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testListDistanceWithSimplifier01, org.simmetrics.builders.StringMetricsTest.CreateForSet#shouldReturnForSet, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testMultisetDistanceWithFilter, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testListMetricWithSimplifier01, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testListDistance02WithCache, org.simmetrics.metrics.StringDistancesTest.CreateStringDistances#jaroWinkler, org.simmetrics.metrics.DamerauLevenshteinTest.OutofBounds#shouldThrowForSubstituteCostBelowBound, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testSetMetricWithSimplifier01WithCache, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testSetDistance, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testMultisetDistanceWithSimplifier01WithCache, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testMultisetMetricWithFilter, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testStringDistance01, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testSetMetric01WithCache, org.simmetrics.simplifiers.SimplifiersTest.ShouldThrowFor#chainWithSingletonList, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testSetMetricWithSimplifier02, org.simmetrics.builders.StringDistancesTest.CreateForMultiset#shouldReturnForSetWithSimplifier, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testListDistanceWithSimplifier01WithCache, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testSetMetric02, org.simmetrics.builders.StringMetricsTest.Create#shouldReturnForListWithChainedSimplifiers, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testMultisetDistance01WithCache, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testMultisetDistanceWithTransform, org.simmetrics.builders.StringMetricsTest.CreateForList#shouldReturnForList, org.simmetrics.metrics.StringMetricsTest.Utilities#blockDistance, org.simmetrics.metrics.StringMetricsTest.CreateStringMetrics#longestCommonSubstring, org.simmetrics.builders.StringDistancesTest.Create#shouldReturnForListWithSimplifier, org.simmetrics.builders.StringDistancesTest.CreateForSet#shouldReturnForSet, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testStringDistanceWithSimplifier02, org.simmetrics.simplifiers.SimplifiersTest.ShouldThrowFor#chainWithNullInVarArg, org.simmetrics.builders.StringMetricsTest.Create#shouldReturnForStringWithChainedSimplifiers, org.simmetrics.builders.StringDistanceBuilderTest.StringDistanceBuilderChainTest#testListDistanceWithSimplifier02, org.simmetrics.builders.StringMetricsTest.CreateForMultiset#shouldReturnForSet, org.simmetrics.metrics.StringDistancesTest.CreateStringDistances#levenshtein, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testListMetricWithSimplifier02WithCache, org.simmetrics.metrics.StringMetricsTest.CreateStringMetrics#damerauLevenshtein, org.simmetrics.simplifiers.SimplifiersTest.ShouldThrowFor#chainWithSingle, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testListMetricWithTransform, org.simmetrics.metrics.HammingDistanceTest.DistanceList#shouldThrowForDifferentLength, org.simmetrics.builders.StringDistancesTest.Create#shouldReturnForString, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testListMetricWithSimplifier02, org.simmetrics.metrics.StringMetricsTest.CreateStringMetrics#levenshtein, org.simmetrics.builders.StringMetricsTest.Create#shouldReturnForString, org.simmetrics.metrics.StringMetricsTest.CreateStringMetrics#jaroWinkler, org.simmetrics.metrics.StringDistancesTest.CreateStringDistances#jaro, org.simmetrics.builders.StringMetricsTest.Create#shouldReturnForSetWithSimplifier, org.simmetrics.builders.StringMetricBuilderTest.StringMetricBuilderChainTest#testMultisetMetricWithSimplifier02WithCache, org.simmetrics.metrics.StringDistancesTest.CreateStringDistances#damerauLevenshtein)
	at edu.illinois.cs.testrunner.runner.SmartRunner.$anonfun$run$1(SmartRunner.scala:40)
	at scala.util.Success.flatMap(Try.scala:247)
	at edu.illinois.cs.testrunner.runner.SmartRunner.run(SmartRunner.scala:33)
	at edu.illinois.cs.testrunner.runner.Runner.runList(Runner.scala:28)
	at edu.illinois.cs.testrunner.runner.Runner.runList$(Runner.scala:27)
	at edu.illinois.cs.testrunner.runner.SmartRunner.runList(SmartRunner.scala:21)
	at edu.illinois.cs.dt.tools.detection.DetectorUtil.originalResults(DetectorUtil.java:32)
	at edu.illinois.cs.dt.tools.detection.detectors.RandomDetector.<init>(RandomDetector.java:26)
	at edu.illinois.cs.dt.tools.detection.detectors.DetectorFactory.makeDetector(DetectorFactory.java:19)
	at edu.illinois.cs.dt.tools.detection.DetectorPlugin.detectorExecute(DetectorPlugin.java:239)
	at edu.illinois.cs.dt.tools.detection.DetectorPlugin.lambda$execute$5(DetectorPlugin.java:201)
	at edu.illinois.cs.dt.tools.utility.ErrorLogger.runAndLogError(ErrorLogger.java:39)
	at edu.illinois.cs.dt.tools.detection.DetectorPlugin.execute(DetectorPlugin.java:201)

Fail to Find JUnit 5 Inherited Tests of Nested Class

Example: CreateIdentity

Fail to Find JUnit 5 Non-public Tests

JUnit 5 tests can be non-public, while the current logic (getMethods) can only return public methods.

One simple solution is to use getDeclaredMethods because it returns all methods declared by a class. However, getDeclaredMethods can't return the methods inherited from the superclass, which means it can't fix the issue - Fail to Find JUnit 5 Tests Inherited from the Superclass.

Failed to Detect Flaky Test

The old version of testrunner(1.0)/iDFlakies(1.0.2) can easily detect some flaky tests on gradle/performance-comparisons. But the latest testrunner(1.1)/iDFlakies(1.1.0) fails to detect those flaky tests.

Project: gradle/performance-comparisons
Branch: commons-lang (there is only one commit)

Flaky tests:

(from list.txt)
org.apache.commons.lang3.time.FastDateFormatTest.test_getInstance_String_Locale
org.apache.commons.lang3.time.FastDateFormatTest.test_changeDefault_Locale_DateTimeInstance
org.apache.commons.lang3.time.FastDateFormatTest.test_changeDefault_Locale_DateInstance
org.apache.commons.lang3.time.FastDatePrinterTest.testTimeZoneAsZ
org.apache.commons.lang3.time.FastDateFormat_PrinterTest.testTimeZoneAsZ
org.apache.commons.lang3.time.DateFormatUtilsTest.testSMTP
org.apache.commons.lang3.time.DateUtilsTest.testLANG799_EN_OK
org.apache.commons.lang3.time.DateUtilsTest.testLANG799_EN_FAIL
org.apache.commons.lang3.time.DateUtilsTest.testLANG799_DE_OK
org.apache.commons.lang3.time.DateUtilsTest.testLANG799_DE_FAIL

Refactor Needed for Tests with @ParameterizedTest Annotation (JUnit 5)

Each @ParameterizedTest test can run several times, based on its parameters. So there may be more than one TestResult generated for each test. Refactor is needed.

The current behavior is to filter out tests with @ParameterizedTest annotation when locating tests.

Fail to Find JUnit 5 Tests Inherited from the Superclass

Update: The below example is not appropriate because it actually has two problems. 1) it inherits test methods from the superclass, 2) it's a nested class. We should expect an example with only the first problem

Example: CreateIdentity

This class inherits many tests from its grandparent class DistanceTest.java. But the testrunner fails to find them, which means we can't find those tests in the generated original-order file.

So after fixing this issue, we should be able to find those tests in the original-order file.