testingresearchillinois / starts Goto Github PK

We have many String literals that are duplicated and scattered all over the codebase, e.g., changed-classes, jar-checksums, all-tests. The goal here is to refactor; move all these literals to edu.illinois.starts.constants.StartsConstants.

I downloaded commons-collections with the latest version whose SHA is 7e953fd6dd0d35e19239169edf72298be02d550e. Then I revised pom.xml and added starts plugin.
The command I used is "mvn starts:starts -DstartsLogging=FINEST -DdepFormat=CLZ -Drat.skip"

java version is 1.8
maven version is 3.6.3
in pom.xml, its jdk version is 1.8

The error message is

Users/home/.m2/repository/org/junit/vintage/junit-vintage-engine/5.6.2/junit-vintage-engine-5.6.2.jar,
/Users/home/.m2/repository/junit/junit/4.13/junit-4.13.jar,
/Users/home/.m2/repository/org/hamcrest/hamcrest/2.2/hamcrest-2.2.jar,
/Users/home/.m2/repository/org/easymock/easymock/4.2/easymock-4.2.jar,
/Users/home/.m2/repository/org/objenesis/objenesis/3.1/objenesis-3.1.jar,
/Users/home/.m2/repository/org/apache/commons/commons-lang3/3.11/commons-lang3-3.11.jar,
/Users/home/.m2/repository/commons-io/commons-io/2.8.0/commons-io-2.8.0.jar,
/Users/home/.m2/repository/commons-codec/commons-codec/1.15/commons-codec-1.15.jar]
FINE: [PROFILE] updateForNextRun(getSureFireClassPath): 0.014 FINE:
[PROFILE] updateForNextRun(pathToString): 0.000 FINEST: @@Excludes:
[**/*$*, **/TestUtils.java, **/Abstract*.java, **/BulkTest.java]
FINEST: @@Includes: [**/*Test.java] FINE: [PROFILE]
updateForNextRun(setIncludesExcludes): 0.001 FINE: [PROFILE]
updateForNextRun(getTestClasses): 0.064 FINE: [PROFILE]
updateForNextRun(createClassLoader): 0.004 FINE: JDEPS ARGS:[-v,
/Users/home/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.6.2/junit-jupiter-api-5.6.2.jar]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE [INFO]
------------------------------------------------------------------------
[INFO] Total time: 20.107 s [INFO] Finished at:
2020-10-20T23:51:33-05:00 [INFO]
------------------------------------------------------------------------
--------------------------------------------------- constituent[0]:
file:/Users/home/apache-maven/conf/logging/ constituent[1]:
file:/Users/home/apache-maven/lib/maven-plugin-api-3.6.3.jar
constituent[2]:
file:/Users/home/apache-maven/lib/maven-compat-3.6.3.jar
constituent[3]:
file:/Users/home/apache-maven/lib/maven-resolver-util-1.4.1.jar
constituent[4]:
file:/Users/home/apache-maven/lib/plexus-component-annotations-2.1.0.jar
constituent[5]:
file:/Users/home/apache-maven/lib/maven-settings-builder-3.6.3.jar
constituent[6]:
file:/Users/home/apache-maven/lib/plexus-cipher-1.7.jar
constituent[7]:
file:/Users/home/apache-maven/lib/plexus-interpolation-1.25.jar
constituent[8]:
file:/Users/home/apache-maven/lib/maven-resolver-spi-1.4.1.jar
constituent[9]:
file:/Users/home/apache-maven/lib/guice-4.2.1-no_aop.jar
constituent[10]:
file:/Users/home/apache-maven/lib/maven-artifact-3.6.3.jar
constituent[11]:
file:/Users/home/apache-maven/lib/maven-resolver-provider-3.6.3.jar
constituent[12]: file:/Users/home/apache-maven/lib/cdi-api-1.0.jar
constituent[13]:
file:/Users/home/apache-maven/lib/maven-slf4j-provider-3.6.3.jar
constituent[14]:
file:/Users/home/apache-maven/lib/plexus-utils-3.2.1.jar
constituent[15]:
file:/Users/home/apache-maven/lib/plexus-sec-dispatcher-1.4.jar
constituent[16]:
file:/Users/home/apache-maven/lib/maven-repository-metadata-3.6.3.jar
constituent[17]:
file:/Users/home/apache-maven/lib/commons-cli-1.4.jar
constituent[18]:
file:/Users/home/apache-maven/lib/maven-resolver-transport-wagon-1.4.1.jar
constituent[19]: file:/Users/home/apache-maven/lib/commons-io-2.5.jar
constituent[20]: file:/Users/home/apache-maven/lib/jansi-1.17.1.jar
constituent[21]:
file:/Users/home/apache-maven/lib/maven-model-builder-3.6.3.jar
constituent[22]:
file:/Users/home/apache-maven/lib/maven-embedder-3.6.3.jar
constituent[23]:
file:/Users/home/apache-maven/lib/wagon-file-3.3.4.jar
constituent[24]:
file:/Users/home/apache-maven/lib/org.eclipse.sisu.inject-0.3.4.jar
constituent[25]:
file:/Users/home/apache-maven/lib/maven-resolver-connector-basic-1.4.1.jar
constituent[26]:
file:/Users/home/apache-maven/lib/maven-settings-3.6.3.jar
constituent[27]:
file:/Users/home/apache-maven/lib/maven-model-3.6.3.jar
constituent[28]:
file:/Users/home/apache-maven/lib/maven-resolver-api-1.4.1.jar
constituent[29]:
file:/Users/home/apache-maven/lib/jcl-over-slf4j-1.7.29.jar
constituent[30]:
file:/Users/home/apache-maven/lib/wagon-http-3.3.4-shaded.jar
constituent[31]:
file:/Users/home/apache-maven/lib/org.eclipse.sisu.plexus-0.3.4.jar
constituent[32]:
file:/Users/home/apache-maven/lib/slf4j-api-1.7.29.jar
constituent[33]:
file:/Users/home/apache-maven/lib/maven-builder-support-3.6.3.jar
constituent[34]: file:/Users/home/apache-maven/lib/jsoup-1.12.1.jar
constituent[35]:
file:/Users/home/apache-maven/lib/wagon-provider-api-3.3.4.jar
constituent[36]:
file:/Users/home/apache-maven/lib/maven-shared-utils-3.2.1.jar
constituent[37]:
file:/Users/home/apache-maven/lib/maven-resolver-impl-1.4.1.jar
constituent[38]:
file:/Users/home/apache-maven/lib/maven-core-3.6.3.jar
constituent[39]: file:/Users/home/apache-maven/lib/javax.inject-1.jar
constituent[40]:
file:/Users/home/apache-maven/lib/commons-lang3-3.8.1.jar
constituent[41]:
file:/Users/home/apache-maven/lib/guava-25.1-android.jar
constituent[42]: file:/Users/home/apache-maven/lib/jsr250-api-1.0.jar
--------------------------------------------------- Exception in
thread "main" java.lang.InternalError: Missing message:
warn.skipped.entry at
com.sun.tools.jdeps.JdepsTask.getMessage(JdepsTask.java:709) at
com.sun.tools.jdeps.JdepsTask.warning(JdepsTask.java:673) at
com.sun.tools.jdeps.JdepsTask.findDependencies(JdepsTask.java:563) at
com.sun.tools.jdeps.JdepsTask.run(JdepsTask.java:324) at
com.sun.tools.jdeps.JdepsTask.run(JdepsTask.java:306) at
com.sun.tools.jdeps.Main.run(Main.java:63) at
edu.illinois.starts.helpers.RTSUtil.runJdeps(RTSUtil.java:78) at
edu.illinois.starts.helpers.Cache.checkMissingJarsInJDKCache(Cache.java:78)
at
edu.illinois.starts.helpers.Cache.loadM2EdgesFromCache(Cache.java:48)
at
edu.illinois.starts.jdeps.BaseMojo.prepareForNextRun(BaseMojo.java:227)
at
edu.illinois.starts.jdeps.DiffMojo.prepareForNextRun(DiffMojo.java:33)
at
edu.illinois.starts.jdeps.DiffMojo.updateForNextRun(DiffMojo.java:103)
at edu.illinois.starts.jdeps.RunMojo.run(RunMojo.java:81) at
edu.illinois.starts.jdeps.RunMojo.execute(RunMojo.java:62) at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
at
org.apache.maven.lifecycle.internal.MojoExecutor.executeForkedExecutions(MojoExecutor.java:355)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:200)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305) at
org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192) at
org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105) at
org.apache.maven.cli.MavenCli.execute(MavenCli.java:957) at
org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289) at
org.apache.maven.cli.MavenCli.main(MavenCli.java:193) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)~

I added the following to the pom.xml on a clean clone of commons-fileupload:

<plugin>
  <groupId>edu.illinois</groupId>
  <artifactId>starts-maven-plugin</artifactId>
  <version>1.4-SNAPSHOT</version>
  <configuration>
    <depFormat>CLZ</depFormat>
  </configuration>
</plugin>

When I run mvn -Drat.skip starts:starts for multiple times, STARTS always selects and runs all tests, instead of selecting nothing.

The goal here is to improve on #13 by

1. adding more ways that the classpath can change (e.g., order of jars can affect which class is loaded)
2. checking if the change to the classpath is really used by the code under test
3. add tests

travis-ci.org will be deprecated.

As discussed in #72

In #13, we remove all SNAPSHOT jars from the classpath before checking. We should only remove SNAPSHOT files that are in the same project (e.g., in a different module).

'mvn starts:starts' failed on NonDex 'nondex-maven-plugin' module because checkstyle is triggered and the check failed the generated sources of the module.
The sha for NonDex I'm working with is a1d2fa21e19acd5453abf4c549b61101ac115e3b.

Currently, mvn starts:select will find tests that are affected by a change, but will not run them. It would be nice if, in addition to the list of affected tests, mvn starts:select also produces an estimate of how long those affected tests would have taken if they were run (e.g., based on how long they took in the past).

in order for integration tests to work AND allow the user to look thru all available artifacts, the cleanup code that does stuff like deletefiles and deletefolders will need to be moved to setuputil and called from setup.groovy. they are currently in verifyutil and called from verify.groovy.

Currently, when retestAll is triggered only by classpath changes, STARTS gives a wrong message, saying 0 tests are affected when all tests are in fact affected:

INFO: STARTS:AffectedTests: 0
INFO: STARTS:TotalTests: 15
INFO: ********** Run **********
INFO: No tests are selected to run.

The file generated here is not incorporating the options for calculating affected classes.

TODO:

• Investigate why that is the case

detailed here: travis-ci/travis-ci#7304

it seems unavoidable. One of my recent PRs was an "enhancement" to the .travis.yml.
I don't think waiting 20-30 minutes for the macOS build to start is worth it. So I can either:

• remove the macOS portion from the .travis.yml build matrix. the .travis.yml is still enhanced compared to before, it just won't build on macOS
• keep the macOS portion, but set it as optional so that build status will not depend on macOS. In other words, as soon as the linux one finishes, the build is complete. macOS will run when it runs, but build status does not wait or depend on macOS.

Or we can keep things as is. Please let me know which you'd prefer, I can make the change.

The attached screenshot below shows that there was a 15 minute delay. I've noticed delays past 30 minutes before.

Concerning pull request #3, we should have some documentation of what different logging levels log.

'mvn starts:starts' failed on NonDex, reporting 'project nondex: Surefire plugin not available' .
The surefire plugin is declared in the pom.xml of NonDex under <project><build><pluginManagement><plugins>.
Adding surefire plugin under <project><build><plugins> fixes this issue.
The sha for NonDex I'm working with is a1d2fa21e19acd5453abf4c549b61101ac115e3b.

I included the plugin xml code as mentioned in the read.me (as well as installed it from the source code) but couldn't get STARTS work with mvn starts:starts. I get the following ERROR -

No plugin found for prefix 'starts' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (/Users/<USERNAME>/.m2/repository), central (https://repo.maven.apache.org/maven2)]

However, I installed it from the source and used the following command instead and it works wonder -
mvn edu.illinois:starts-maven-plugin:1.4-SNAPSHOT:starts

Thought, you guys should know and update the read.me .

When retestAll is false, getJarToChecksumMapping will be called in RunMojo, and it throws an exception.

Reproduce: After the initial run mvn starts:starts which runs all tests, modify some code of the project being built, so that STARTS is supposed to run a subset of the tests in the next build. Then run mvn starts:starts again, and I can get the error below.

java.io.IOException: Is a directory
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:159)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at edu.illinois.starts.helpers.Writer.getJarToChecksumMapping(Writer.java:233)
at edu.illinois.starts.jdeps.RunMojo.hasSameJarChecksum(RunMojo.java:161)
at edu.illinois.starts.jdeps.RunMojo.run(RunMojo.java:95)
at edu.illinois.starts.jdeps.RunMojo.execute(RunMojo.java:82)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.MojoExecutor.executeForkedExecutions(MojoExecutor.java:365)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:199)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)

@owolabileg Hi Professor Legunsen :
Is there a way to make STARTS looking for test classes in a folder other than test-classes?
I want to do this because right now I am using defects4j's pipeline and it compiles a project's tests into target/tests. I only want to use STARTS to pick the affected tests for me, and right now STARTS seems to be reading tests from target/test-classes.
Is there a way to do this by making changes to STARTS?

What I tried

• I looked into STARTS's code and found the constant TEST_CLASSES, modifying it gave me no luck.
• I tried to give sure-fire parameter through command line, for example,

mvn edu.illinois:starts-maven-plugin:1.4-SNAPSHOT:select -Dmaven.test.skip=true  -DtestClassesDirectory=$(pwd)/target/tests -X

Both gave me no luck.

All the integration tests currently fail like so:

[INFO] Building: base-it\pom.xml
[DEBUG] Build log initialized in: C:\Users\owolabi\projects\starts\starts-plugin\target\it\base-it\build.log
[DEBUG] no script found in directory: C:\Users\owolabi\projects\starts\starts-plugin\target\it\base-it
[DEBUG] no script found in directory: C:\Users\owolabi\projects\starts\starts-plugin\target\it\base-it
[DEBUG] Using MAVEN_OPTS: null
[DEBUG] Executing: cmd.exe /X /C "C:\cygwin64\home\owolabi\Software\apache-maven-3.5.0\bin\mvn.cmd -B -D maven.repo.local=C:\Users\owolabi\projects\starts\starts-plugin\target\local-repo -s C:\Users\owolabi\projects\starts\starts-plugin\target\it\interpolated-settings.xml package"
[INFO] ..FAILED (0.0 s)
[INFO]   The Maven invocation failed. Error while executing process.

Do we have support of STARTS for scala projects using sbt? I tried adding following sbt dependencies to scala project, ran scalatests but couldn't run it using sbt command.

https://mvnrepository.com/artifact/edu.illinois/starts/1.3

Can someone please tell me what is the correct way to run the tests?

Hi:
I am trying to integrate STARTS with defects4j, which is a project containing real-world java projects and real faults. To make defects4j work properly, it is suggested I use java 7. So what I do is this:

• I checkout a new, uncompiled version using defects4j command
• I run mvn starts:select to output the list of impacted tests.
  Here comes the error:

[ERROR] Failed to execute goal edu.illinois:starts-maven-plugin:1.3:select (default-cli) on project commons-lang3: 
Execution default-cli of goal edu.illinois:starts-maven-plugin:1.3:select failed: 
An API incompatibility was encountered while executing edu.illinois:starts-maven-plugin:1.3:select: java.lang.UnsupportedClassVersionError: edu/illinois/yasgl/DirectedGraph : Unsupported major.minor version 52.0

• echo $JAVA_HOME : /usr/lib/jvm/jdk1.7.0_80
• java -version: java version "1.7.0_80"

Previously, I use java 8 and everything about STARTS was fine, however, there appears that some of the developer-written test in defects4j will broken and turned out failed.
Is there any suggestion here ?

Regards,
Zhenming

Summary: If one uses @RunWith(Suite.class), dependencies do not
include code under test (or test classes that are in the suite). This
is a safety issue, i.e., STARTS may not select a test with changed
behavior.

Steps to Reproduce: Unzip the attached file. Run the bash
script ./s provided in this directory. This script first runs test with
STARTS, then change the return value of a method under test,
finally runs the same test with STARTS again.

Expected Results: p.RegressionTest runs in both the first and the
second execution. It passes in the first execuion, and it fails in
the second execution.

Actual Results: p.RegressionTest runs only in the first execution.
The test is not selected in the second execution.

starts_bug_test_suite.zip

We need to replace patterns like this: String path = "a" + File.separator + "b" with String path = Path.get("a", "b") for reasons that were discussed in #13

@jtx1999 this is a continuation of #73

Currently, STARTS will throw an exception whenever a project uses versions of surefire that are below 2.13.

Instead of throwing an exception, STARTS should print a very visible warning and then run all tests.