The maven versions plugin has a versions:compare-dependencies mojo that compares "dependency versions of the current project to dependencies or dependency management of a remote repository project. Can optionally update locally the project instead of reporting the comparison"

https://www.mojohaus.org/versions-maven-plugin/compare-dependencies-mojo.html

This may be a way projects can upgrade and test against our BOMs without importing them.

and make DependencyGraph constructor package private.

seems to be a bug in the last PR or 2 such that we're no longer picking up the managed deps:

$ ls /Users/elharo/cloud-opensource-java/target/dashboard
com.google.cloud_cloud-oss-bom_0.62.0-SNAPSHOT.html
dashboard.html

Might be able to use

https://maven.apache.org/ref/3.5.4/maven-resolver-provider/apidocs/org/apache/maven/repository/internal/MavenWorkspaceReader.html

and Maven internal implementation of this. Maven doesn't expose a public API for this.

I.e. devs who are writing or updating the BOM. E.g. how to bump a version in the first or second order dependencies.

"This most commonly happens when a library has published the same classes in artifacts with different group ID or artifact IDs."

I suspect this may be even more common when a third party has forked and released a Maven artifact with a different group ID but the same package. I've seen this happen with Guava for example.

Instead of cloud-opensource-java

E.g. all deps flow from BOM
Minimize dependencies

There's something in CT4E we can copy.

Machine consumable file (likely Maven pom.xml) listing:

The immediate direct dependencies and versions of the libraries listed in #16

Test to verify this.

It hangs on the call to DependencyGraphBuilder.getCompleteDependencies: https://github.com/GoogleCloudPlatform/cloud-opensource-java/blob/master/dependencies/src/test/java/com/google/cloud/tools/opensource/dependencies/DependencyGraphBuilderTest.java#L50

As a first step we can put this on the dashboard. In particular:

1. Create a master list of all the dependency upgrade paths across all the libraries.
2. Remove duplicates.
3. Filter out any upgrades made moot by other upgrades.
4. Possibly make sure dependencies on other things in the BOM are at BOM version.

Maybe even as a first pass do direct dependencies only with a require upper bounds type check across the whole BOM tree (minus testlibs).

Put it all on the dashboard.

I.e. specify the dependency element that should be added to the top level pom.xml in either dependencies or dependencyManagement section.

That is, something much like mvn dependency:tree except:

1. Project may not be checked out or have source code available.
2. Project may be built with gradle, ant, or bazel instead of maven.

That is, reads metadata from central repo instead of local filesystem.

Our code reports that com.google.cloud:google-cloud-bom:0.61.0-alpha has 172 managed dependencies:

  @Test
  public void tesReadBom() throws ArtifactDescriptorException {
    Artifact artifact = new DefaultArtifact("com.google.cloud:google-cloud-bom:0.61.0-alpha");
    List<Artifact> managedDependencies = RepositoryUtility.readBom(artifact);
    // Characterization test. As long as the artifact doesn't change (and it shouldn't)
    // the answer won't change.
    Assert.assertEquals(172, managedDependencies.size());
  }

I don't buy it. That's far too many. Most likely there's something we don't understand here. I'm not sure where the misunderstanding lies: in how a managedDependencies section is interpreted in Maven pom.xml? What Aether's ArtifactDescriptorResult.getManagedDependencies() method is reporting? Something else?

There's something fishy here.

dependency upgrade tool done.
Need to pull dependencies from BOM.

Maybe make it black?

instead of System.err.println()

When working with IntelliJ and Git, it's convenient to exclude .idea and *.iml so that we don't have to worry about them unexpectedly added to staging area.

Ref https://github.com/GoogleContainerTools/jib/blob/master/.gitignore

We can use the Maven resolver dependencies library instead.

figure out why this happens:

First question: when does this happen? Is this the output of our app, or something that happens on writes to GCS, or on reads from GCS?

{"error":{"errors":[{"domain":"global","reason":"badLockedDomainRequest","message":"Bad Locked Domain Request","debugInfo":"Error extracting locked domain from query string\ncom.google.api.server.core.Fault: ImmutableErrorDefinition{base=BAD_LOCKED_DOMAIN_REQUEST, category=USER_ERROR, cause=null, debugInfo=Error extracting locked domain from query string, domain=global, extendedHelp=null, httpHeaders={}, httpStatus=badRequest, internalReason=Reason{arguments={}, cause=null, code=null, createdByBackend=false, debugMessage=null, errorProtoCode=null, errorProtoDomain=null, filteredMessage=null, location=null, message=null, unnamedArguments=[]}, location=null, message=Bad Locked Domain Request, reason=badLockedDomainRequest, rpcCode=400} Bad Locked Domain Request: Error extracting locked domain from query string\n\tat com.google.api.server.responsesafety.ApiRequestUriCrypter.getEncryptedApiRequest(ApiRequestUriCrypter.java:197)\n\tat com.google.api.server.responsesafety.ApiRequestUriCrypter.unlock(ApiRequestUriCrypter.java:107)\n\tat com.google.api.server.responsesafety.LockedStrategy.lockedDomainInterceptorRequest(LockedStrategy.java:81)\n\tat com.google.api.server.responsesafety.LockedDomainInterceptor.processRequest(LockedDomainInterceptor.java:59)\n\tat com.google.api.server.core.intercept.AroundInterceptorWrapper.processRequest(AroundInterceptorWrapper.java:20)\n\tat com.google.api.server.stats.StatsBootstrap$InterceptorStatsRecorder.processRequest(StatsBootstrap.java:278)\n\tat com.google.api.server.core.intercept.Interceptions$AroundInterception.processRequest(Interceptions.java:159)\n\tat com.google.api.server.core.intercept.Interceptions$AroundInterception.invoke(Interceptions.java:135)\n\tat com.google.api.server.media.AbstractMediaClosure.continueRequest(AbstractMediaClosure.java:64)\n\tat com.google.api.server.media.batch.BatchAgent.continueRequest(BatchAgent.java:141)\n\tat com.google.api.server.media.batch.BatchAgent.onRequestReceived(BatchAgent.java:74)\n\tat com.google.uploader.agent.ScottyBatchAgent$ServiceParameters$1.handleRequest(ScottyBatchAgent.java:418)\n\tat com.google.net.rpc3.impl.server.RpcServerInternalContext.runRpcInApplication(RpcServerInternalContext.java:577)\n\tat com.google.net.rpc3.impl.server.RpcServerChannel$1.apply(RpcServerChannel.java:879)\n\tat com.google.net.rpc3.impl.server.RpcServerChannel$1.apply(RpcServerChannel.java:867)\n\tat com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:240)\n\tat com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:230)\n\tat com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:117)\n\tat com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:455)\n\tat com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:462)\n\tat com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:323)\n\tat com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:313)\n\tat com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:459)\n\tat com.google.common.util.concurrent.MoreExecutors$5$1.run(MoreExecutors.java:1056)\n\tat com.google.api.server.thread.ThreadTrackers$ThreadTrackingRunnable.run(ThreadTrackers.java:126)\n\tat com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:455)\n\tat com.google.api.server.server.CommonModule$ContextCarryingExecutorService$1.runInContext(CommonModule.java:846)\n\tat com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:462)\n\tat com.google.tracing.CurrentContext.runInContext(CurrentContext.java:320)\n\tat com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:321)\n\tat com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:313)\n\tat com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:459)\n\tat com.google.gse.internal.DispatchQueueImpl$WorkerThread.run(DispatchQueueImpl.java:403)\n"}],"code":400,"message":"Bad Locked Domain Request"}}

The dependencies library at least could use this

So we can see what it looks like and configure the bucket to serve the static HTML

could even autosubmit patches

• dependency:analyze
  • This can detect unused dependencies
• versions
  • This can help find & update to the newest versions of dependencies
• checkstyle
  • Help find banned imports
• compatibility
  • Clirr
  • japi-compliance-checker

Machine consumable file (likely Maven pom.xml) listing:

The complete dependency list of the libraries in #17 along with their required versions.

Test to verify this.

The BOM needs a test that all the artifacts referenced are in Maven Central.

Just realized I incorrectly hyphenated one. Found it in another test but there should be a direct test for this. The libraries.json tests in appengine-plugins-core have code like this.

That is instead of

https://storage.googleapis.com/cloud-opensource-java-dashboard/prod/cloud-opensource-java/ubuntu/periodic/7/20180715-103104/dashboard/target/dashboard/dashboard.html

where the path components "7/20180715-103104/" change with every build, we need a stable URL such as

https://storage.googleapis.com/cloud-opensource-java-dashboard/dashboard/dashboard.html

For generating the dependency upgrade todo list.

i.e. devs who simply import the libraries

E..g com.google.foo needs to upgrade jsr305 to 3.0.2
...

start at the bottom of the dependency tree and work our way up.
How much of this can be farmed out?
Can this be autogenerated from the zeroth order dependency list?

This needs the zeroth order dependency list first.

see #30

• build a test to check the status
• find a way to actually display the status

Consider whether we want to test the latest release in the dashboard or a specific version

See #37

at root level with submodules

1. Are there migration instructions?
2. Are there reasons other than inertia projects might want to stick with
   google-oauth-java-client?
3. Should we formally mark
   google-oauth-java-client as deprecated, and if not, why not?

Machine consumable file (likely Maven pom.xml) listing:

All immediate libraries we commit to supporting along with versions that work together. E.g. most google-cloud-java libraries plus a few others that aren't yet pulled into google-cloud-java.

Should be in a new top-level directory.

Figure out how to do this

I'm having trouble parsing this:

Scrutinize all dependency additions. Check the result of mvn dependency:tree (after running mvn install -DskipTests to build the library) to see which transitive dependencies are added by just adding a single dependency to your own library, and if you require all of the transitive dependencies

It's also unclear how exactly this applies to non-Maven projects. I think I get the intent, but let's take another pass at the language used to describe this.

E.g. start with beam; find the full transitive dependency graph; identify all dependencies with multiple versions; and identify the projects and artifacts that need to be updated to later versions.

Maybe even autogenerate PRs fr github.

waiting on master pom

We need a tool that can verify that a single artifact is linkage compatible. This should verify that all static references in a given classpath are valid. This is a necessary tool to verify that we're actually meeting the objective of not surfacing conflicts to users. (Verifying version agreement is not sufficient.)

The tool takes the Maven coordinates?/pom.xml file? as input. It resolves the dependency tree according to the usual Maven dependency mediation algorithm. Thus all artifacts are assigned specific versions. This is the runtime classpath. Then it checks this for linkage conflicts.

Linkage conflict: The signature of a non-private method or class in a dependency has changed in an incompatible way between the version supplied at compile time and the version invoked at runtime. For example, a public method may be removed from a class or a class may be made final. Linkage conflicts detected at runtime manifest as ReflectiveOperationException, NoClassDefFoundError, NoSuchFieldException, MethodNotFoundException, LinkageError, or other related exceptions.
Or, another perspective: In cases where binary compatibility and source compatibility are the same, a linkage conflict is when compilation would fail if the libraries in the classpath were all built together from their originating source code, or when reflection would fail.
Opposite: Linkage-compatible.
Sub-type: Static linkage conflict: A linkage conflict caused by a direct code reference (non-reflective).
Sub-type: Reflective linkage conflict: A linkage conflict caused by a reflective reference.

It runs through the classpath, finds every statically referenced class, interface, method, and field and verifies the signatures match up. E.g. no signature has changed or been removed in an incompatible way in the specific version found in the classpath.

We are not considering access via reflection.

Implementation: maven plugin? main() method? dashboard report? all of this?

Start with a one page design doc?

How does this differ from simple compilation checking?
Are these errors common in practice?

@garrettjonesgoogle Can you flesh this issue out a bit?

Per https://maven.apache.org/resolver/apidocs/org/eclipse/aether/util/graph/transformer/ConflictResolver.html and https://github.com/apache/maven-resolver/blob/84a32a866ff27d4df75124074fd925588f4a574d/maven-resolver-demos/maven-resolver-demo-snippets/src/main/java/org/apache/maven/resolver/examples/util/Booter.java

it may be possible to have Maven retain conflicting dependencies like so:

    // uncomment to generate dirty trees
    // session.setDependencyGraphTransformer( null );

This might be faster than what we're doing.

Based ons status of individual libraries

Not the same as scope in maven terms

to avoid Chrome trying to translate it