bazelbuild / migration-tooling Goto Github PK

View Code? Open in Web Editor NEW

44.0 44.0 30.0 11.1 MB

Migration tools for Bazel

License: Apache License 2.0

Python 13.51% Java 86.49%

bazel java

migration-tooling's People

Contributors

Stargazers

Watchers

migration-tooling's Issues

WARNING: Unable to resolve version

I get WARNING: Unable to resolve version on a handful of maven jars. Here's an example:

% bazel run //generate_workspace -- --artifact=jaxen:jaxen:1.1.6 
INFO: Found 1 target...
Target //generate_workspace:generate_workspace up-to-date:
  bazel-bin/generate_workspace/generate_workspace.jar
  bazel-bin/generate_workspace/generate_workspace
INFO: Elapsed time: 0.065s, Critical Path: 0.00s

INFO: Running command line: bazel-bin/generate_workspace/generate_workspace '--artifact=jaxen:jaxen:1.1.6'
Nov 15, 2017 12:11:59 AM com.google.devtools.build.workspace.maven.Resolver resolveArtifact
WARNING: Unable to resolve version
Wrote /home/admin/.cache/bazel/_bazel_admin/656746585370bf916b19c45e0d595b0a/execroot/__main__/bazel-out/local-fastbuild/bin/generate_workspace/generate_workspace.runfiles/__main__/generate_workspace.bzl

% cat /home/admin/.cache/bazel/_bazel_admin/656746585370bf916b19c45e0d595b0a/execroot/__main__/bazel-out/local-fastbuild/bin/generate_workspace/generate_workspace.runfiles/__main__/generate_workspace.bzl
# The following dependencies were calculated from:
#
# generate_workspace --artifact jaxen:jaxen:1.1.6


def generated_maven_jars():
  pass



def generated_java_libraries():
  pass

The library in question is at https://search.maven.org/#artifactdetails%7Cjaxen%7Cjaxen%7C1.1.6%7Cbundle

Update Transitive_maven_jar rule README to include repositories field

Currently following the README errors complaining about repositories option missing.

Fix by adding repository field to set the option
example:

transitive_maven_jar(
name = "dependencies",
artifacts = [
"org.springframework.cloud:spring-cloud-config-server:1.3.0.RELEASE",
],
repositories = ["",],
)

Fails with google-cloud-storage (version missing)

I'm getting this error when trying to fetch my dependencies when I include com.google.cloud:google-cloud-storage (pom.xml):

com.google.cloud:google-cloud-storage scope: compile
Sep 07, 2017 12:14:08 AM com.google.devtools.build.workspace.maven.DefaultModelResolver getEffectiveModel
WARNING: Unable to resolve Maven model from https://repo1.maven.org/maven2/com/google/cloud/google-cloud-storage/1.4.0/google-cloud-storage-1.4.0.pom: 2 problems were encountered while building the effective model for com.google.cloud:google-cloud-storage:1.4.0
[ERROR] 'dependencies.dependency.version' for org.easymock:easymock:jar is missing. @ 
[ERROR] 'dependencies.dependency.version' for org.objenesis:objenesis:jar is missing. @

This is my pom.xml file:

<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                      http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>jigsaw.google.com</groupId>
  <artifactId>ooni-bq-third-party</artifactId>
  <version>0</version>

  <dependencies>
    <!-- As per https://beam.apache.org/get-started/downloads/ -->
    <dependency>
      <groupId>org.apache.beam</groupId>
      <artifactId>beam-sdks-java-core</artifactId>
      <version>2.1</version>
    </dependency>
    <dependency>
      <groupId>org.apache.beam</groupId>
      <artifactId>beam-runners-direct-java</artifactId>
      <version>2.1</version>
    </dependency>
    <dependency>
      <groupId>org.apache.beam</groupId>
      <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
      <version>2.1</version>
    </dependency>

    <!-- For JSON processing -->
    <dependency>
      <groupId>com.google.code.gson</groupId>
      <artifactId>gson</artifactId>
      <version>2.8</version>
    </dependency>

    <!-- For Google Cloud Storage -->
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-storage</artifactId>
      <version>1.4.0</version>
    </dependency>

    <!-- Get proper logging and fix http://www.slf4j.org/codes.html#StaticLoggerBinder -->
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-jdk14</artifactId>
      <version>1.7</version>
    </dependency>
  </dependencies>
</project>

Support for Test and Runtime Maven Scopes

Within the Resolver class (com.google.devtools.build.workspace.maven package), the resolveEffectiveModel function simply skips over any dependencies that are not scoped as COMPILE. Maven supports five other types of dependencies, of particular note are the test and runtime scope. It would be useful to add support to these scopes as well.

Here is the relevant portion of the Resolver

public Model resolveEffectiveModel(ModelSource modelSource, Set<String> exclusions, Rule parent) {
    Model model = modelResolver.getEffectiveModel(modelSource);
    ...
    for (Dependency dependency : model.getDependencies()) {
      if (!dependency.getScope().equals(COMPILE_SCOPE)) {
        continue;
      }
      ...
    }
    ...
}

No repositories searched by default

The commit to enable custom repositories (a5e6874) seems to have caused this project to use no repositories by default. Can we return to using default repositories if no repositories are specified?

% bazel run //generate_workspace -- --artifact junit:junit:4.11 --output_dir /tmp
INFO: Analysed target //generate_workspace:generate_workspace (0 packages loaded).
INFO: Found 1 target...
Target //generate_workspace:generate_workspace up-to-date:
  bazel-bin/generate_workspace/generate_workspace.jar
  bazel-bin/generate_workspace/generate_workspace
INFO: Elapsed time: 0.585s, Critical Path: 0.50s
INFO: Build completed successfully, 3 total actions

INFO: Running command line: bazel-bin/generate_workspace/generate_workspace --artifact junit:junit:4.11 --output_dir /tmp
Feb 16, 2018 7:44:16 PM com.google.devtools.build.workspace.maven.Resolver resolveArtifact
WARNING: Could not find any repositories that knew how to resolve junit:junit:4.11 (checked )
Wrote /tmp/generate_workspace.bzl

% cat /tmp/generate_workspace.bzl
# The following dependencies were calculated from:
#
# generate_workspace --artifact junit:junit:4.11 --output_dir /tmp


def generated_maven_jars():
  pass



def generated_java_libraries():
  pass

If I check out an older commit, it works:

% git co 0f25a7e83f2f4b776fad9c8cb929ec9fa7cac87f
Previous HEAD position was a5e6874... Enable custom repositories. (#71)
HEAD is now at 0f25a7e... Switch output from System.out to System.err (#72)

% bazel run //generate_workspace -- --artifact junit:junit:4.11 --output_dir /tmp
INFO: Analysed target //generate_workspace:generate_workspace (1 packages loaded).
INFO: Found 1 target...
Target //generate_workspace:generate_workspace up-to-date:
  bazel-bin/generate_workspace/generate_workspace.jar
  bazel-bin/generate_workspace/generate_workspace
INFO: Elapsed time: 1.159s, Critical Path: 1.06s
INFO: Build completed successfully, 7 total actions

INFO: Running command line: bazel-bin/generate_workspace/generate_workspace --artifact junit:junit:4.11 --output_dir /tmp
Feb 16, 2018 7:45:06 PM com.google.devtools.build.workspace.maven.Resolver traverseDeps
INFO:   Downloading pom for junit:junit:4.11
Feb 16, 2018 7:45:06 PM com.google.devtools.build.workspace.maven.Resolver traverseDeps
INFO:   Downloading pom for org.hamcrest:hamcrest-core:1.3
Wrote /tmp/generate_workspace.bzl

% cat /tmp/generate_workspace.bzl
# The following dependencies were calculated from:
#
# generate_workspace --artifact junit:junit:4.11 --output_dir /tmp


def generated_maven_jars():
  # junit:junit:jar:4.11
  native.maven_jar(
      name = "org_hamcrest_hamcrest_core",
      artifact = "org.hamcrest:hamcrest-core:1.3",
      sha1 = "42a25dc3219429f0e5d060061f71acb49bf010a0",
  )


  native.maven_jar(
      name = "junit_junit",
      artifact = "junit:junit:4.11",
  )




def generated_java_libraries():
  native.java_library(
      name = "org_hamcrest_hamcrest_core",
      visibility = ["//visibility:public"],
      exports = ["@org_hamcrest_hamcrest_core//jar"],
  )


  native.java_library(
      name = "junit_junit",
      visibility = ["//visibility:public"],
      exports = ["@junit_junit//jar"],
      runtime_deps = [
          ":org_hamcrest_hamcrest_core",
      ],
  )

Gradle support for generating maven_jar rules

Migrated from issue 562 in main repository

It would be nice if there were either a script or built in support to generate maven_jar rules from build.gradle's dependencies closure.

File repositories don't work

A repository pointing at the filesystem fails to resolve any dependencies, due to some assumptions made about POM file locations.

I have a fix for this (although it only works for absolute-path repositories per issue #63).

--artifact is pulling in unrelated artifacts

I ran this on a fresh clone of migration-tooling:

bazel run //generate_workspace -- --output_dir /tmp --artifact=com.google.guava:guava:22.0

and it wrote a generate_workspace.bzl with 138 dependencies:

% grep -A 1 native.maven_jar /tmp/generate_workspace.bzl | grep name | wc -l       
138

% grep -A 1 native.maven_jar /tmp/generate_workspace.bzl | grep name | sort | head
      name = "ant_ant",
      name = "asm_asm",
      name = "backport_util_concurrent_backport_util_concurrent",
      name = "cglib_cglib",
      name = "classworlds_classworlds",
      name = "com_google_code_findbugs_jsr305",
      name = "com_google_errorprone_error_prone_annotations",
      name = "com_google_guava_guava",
      name = "com_google_j2objc_j2objc_annotations",
      name = "com_jcraft_jsch",

When I run the similar command in the main bazel repo, I only get 5:

% cd /tmp/bazel

% bazel run //src/tools/generate_workspace -- --output_dir /tmp --artifact=com.google.guava:guava:22.0

% grep -A 1 maven_jar /tmp/WORKSPACE | grep name | wc -l                                                                                                                       
5

% bazel version
Build label: 0.5.2
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Jun 27 13:27:03 2017 (1498570023)
Build timestamp: 1498570023
Build timestamp as int: 1498570023

Update documentation for how to pass parameters

The README states that one passes parameters to this tool by performing

bazel run //generate_workspace -- --maven_project=/path/to/maven/project

However when doing so, jcommander throws the following exception

INFO: Running command line: bazel-bin/generate_workspace/generate_workspace '--artifact=com.treasuredata.client:td-client:0.7.6'
Exception in thread "main" com.beust.jcommander.ParameterException: Was passed main parameter '--artifact=com.treasuredata.client:td-client:0.7.6' but no main parameter was defined in your arg class
	at com.beust.jcommander.JCommander.getMainParameter(JCommander.java:891)
	at com.beust.jcommander.JCommander.parseValues(JCommander.java:707)
	at com.beust.jcommander.JCommander.parse(JCommander.java:312)
	at com.beust.jcommander.JCommander.parse(JCommander.java:291)
	at com.google.devtools.build.workspace.GenerateWorkspace.main(GenerateWorkspace.java:48)
ERROR: Non-zero return code '1' from command: Process exited with status 1.

The correct manner to pass parameters is:

bazel run //generate_workspace:generate_workspace -- \
      --artifact "com.treasuredata.client:td-client:0.7.6"

The distinction is that one does not use an equals sign. This change came about due to the transition to jcommander to parse command line parameters.

Cannot interpolate variables

It appears this tool has an issue with variable interpolation from properties.

As an example when I run the following command it does not include a reference to OpenCensus in the resulting generate_workspace.bzl;

bazel run //generate_workspace -- -r http://uk.maven.org/maven2 -a org.apache.beam:beam-sdks-java-core:2.6.0 -a org.apache.beam:beam-runners-direct-java:2.6.0 -a org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.6.0 -o ~/workspace/go/src/github.com/connectedventures/bquploader/

<!-- snip -->

INFO: 	Downloading pom for com.google.cloud.bigtable:bigtable-protos:1.0.0-pre3
Sep 17, 2018 4:39:24 PM com.google.devtools.build.workspace.maven.DefaultModelResolver getEffectiveModel
WARNING: Unable to resolve Maven model from https://repo.maven.apache.org/maven2/com/google/cloud/bigtable/bigtable-client-core/1.0.0/bigtable-client-core-1.0.0.pom: 1 problem was encountered while building the effective model for com.google.cloud.bigtable:bigtable-client-core:1.0.0-pre3
[ERROR] 'dependencies.dependency.version' for io.opencensus:opencensus-contrib-grpc-util:jar must be a valid version but is '${opencensus.version}'. @ 

Wrote /Users/nathanfisher/workspace/go/src/github.com/connectedventures/bquploader/generate_workspace.bzl

generate_workspace should/could choose highest maven version number

Migrated from issue 1573 in main repository

If generate_workspace is run with artifacts that depend on a common artifact, it will choose the version from whichever it encounters first. Alternately, it would be better (I think) if it instead chose the highest version. For example:

% bazel run //src/tools/generate_workspace -- \
  --artifact=com.amazonaws:aws-java-sdk-core:jar:1.11.10 \
  --artifact=io.dropwizard:dropwizard-core:jar:1.0.0

Generates:

 # com.fasterxml.jackson.datatype:jackson-datatype-guava:bundle:2.7.6 wanted version 2.7.6
 # io.dropwizard:dropwizard-jackson:jar:1.0.0 wanted version 2.7.6
 # com.fasterxml.jackson.datatype:jackson-datatype-jsr310:bundle:2.7.6 wanted version 2.7.6
 # com.fasterxml.jackson.module:jackson-module-afterburner:bundle:2.7.6 wanted version 2.7.6
 # com.amazonaws:aws-java-sdk-core:jar:1.11.10
 # com.fasterxml.jackson.datatype:jackson-datatype-jdk8:bundle:2.7.6 wanted version 2.7.6
 maven_jar(
     name = "com_fasterxml_jackson_core_jackson_databind",
     artifact = "com.fasterxml.jackson.core:jackson-databind:2.6.6",
     sha1 = "5108dde6049374ba980b360e1ecff49847baba4a",
 )

But if we reverse the artifacts:

% bazel run //src/tools/generate_workspace -- \
  --artifact=io.dropwizard:dropwizard-core:jar:1.0.0 \
  --artifact=com.amazonaws:aws-java-sdk-core:jar:1.11.10

Now we get:

# com.fasterxml.jackson.datatype:jackson-datatype-guava:bundle:2.7.6
# com.fasterxml.jackson.module:jackson-module-afterburner:bundle:2.7.6
# com.amazonaws:aws-java-sdk-core:jar:1.11.10 wanted version 2.6.6
# io.dropwizard:dropwizard-jackson:jar:1.0.0
# com.fasterxml.jackson.datatype:jackson-datatype-joda:bundle:2.7.6
# com.fasterxml.jackson.datatype:jackson-datatype-jsr310:bundle:2.7.6
# com.fasterxml.jackson.datatype:jackson-datatype-jdk8:bundle:2.7.6
maven_jar(
    name = "com_fasterxml_jackson_core_jackson_databind",
    artifact = "com.fasterxml.jackson.core:jackson-databind:2.7.6",
    sha1 = "c7012a59b4f36843489753a2027e169c3b8586f8",
)

Choosing the highest would also ensure the output is the same regardless of which ordering is used for the artifacts.

For context, this is coming out of a desire to manage our dependencies with a simple list of maven artifacts, and then generating the WORKSPACE and BUILD files automatically. Currently, it seems like either we have to be careful with the ordering of our list of artifacts, or we have to modify the generated WORKSPACE after the fact.

generate_workspace with `--artifact` arg sometimes results in duplicate exports in generated rules

Migrated from issue 3089 in main repository.

Description of the problem

Sometimes the generate_workspace tool generates a BUILD file that fails to build, complaining about a duplicate export label.

e.g. the artifact io.grpc:grpc-protobuf:1.3.0 results in the error

Label '@io_grpc_grpc_protobuf//jar:jar' is duplicated in the 'exports' attribute of rule 'io_grpc_grpc_protobuf'.

If possible, provide a minimal example to reproduce the problem:

To reproduce this problem, first build the generate_workspace tool from the bazel repo (I built from tag 0.4.5 @ 037b9b9).

Then run (in an empty directory):

$ /path/to/generate_workspace -o $(pwd) --artifact io.grpc:grpc-protobuf:1.3.0
ERROR: Could not resolve dependency io.netty:netty-codec-http2:[4.1.3.Final]: Could not find any repositories that knew how to resolve io.netty:netty-codec-http2:[4.1.3.Final] (checked https://oss.sonatype.org/content/repositories/snapshots, https://repo1.maven.org/maven2/)
Wrote:
/Users/jguenther/Projects/tmp/WORKSPACE
/Users/jguenther/Projects/tmp/BUILD

(It looks like the Could not resolve dependency error has already been reported in #1794. We've been working around it by manually removing the square brackets in the version spec, which has worked for us so far)

Running bazel build now fails with the duplicate label error:

$ bazel build //...
ERROR: /Users/jguenther/Projects/tmp/BUILD:122:1: Label '@io_grpc_grpc_protobuf//jar:jar' is duplicated in the 'exports' attribute of rule 'io_grpc_grpc_protobuf'.
ERROR: package contains errors: .
ERROR: error loading package '': Package '' contains errors.
INFO: Elapsed time: 0.136s

And here is the relevant rule from the generated BUILD file:

java_library(
    name = "io_grpc_grpc_protobuf",
    visibility = ["//visibility:public"],
    exports = [
        "@io_grpc_grpc_protobuf//jar",  # here
        "@aopalliance_aopalliance//jar",
        "@com_google_api_api_common//jar",
        "@com_google_api_grpc_grpc_google_common_protos//jar",
        "@com_google_auth_google_auth_library_credentials//jar",
        "@com_google_auto_value_auto_value//jar",
        "@com_google_code_findbugs_jsr305//jar",
        "@com_google_code_gson_gson//jar",
        "@com_google_errorprone_error_prone_annotations//jar",
        "@com_google_guava_guava//jar",
        "@com_google_inject_guice//jar",
        "@com_google_instrumentation_instrumentation_api//jar",
        "@com_google_protobuf_nano_protobuf_javanano//jar",
        "@com_google_protobuf_protobuf_java//jar",
        "@com_google_protobuf_protobuf_java_util//jar",
        "@com_google_protobuf_protobuf_lite//jar",
        "@com_squareup_okhttp_okhttp//jar",
        "@com_squareup_okio_okio//jar",
        "@io_grpc_grpc_all//jar",
        "@io_grpc_grpc_auth//jar",
        "@io_grpc_grpc_context//jar",
        "@io_grpc_grpc_core//jar",
        "@io_grpc_grpc_netty//jar",
        "@io_grpc_grpc_okhttp//jar",
        "@io_grpc_grpc_protobuf//jar",  # and here
        "@io_grpc_grpc_protobuf_lite//jar",
        "@io_grpc_grpc_protobuf_nano//jar",
        "@io_grpc_grpc_stub//jar",
        "@javax_inject_javax_inject//jar",
        "@joda_time_joda_time//jar",
    ],
)

Environment info

Operating System: MacOS El Capitan 10.11.6

Bazel version (output of bazel info release): release 0.4.5-homebrew

(generate_workspace is not included in this distribution, so I built it from the 0.4.5 tag @ 037b9b9)

Have you found anything relevant by searching the web?

(e.g. StackOverflow answers, GitHub issues, email threads on the bazel-discuss Google group)

I looked but didn't find anything relevant to this issue.

Anything else, information or logs or outputs that would be helpful?

An archive of the generated BUILD and WORKSPACE files is attached

Consider implementing migration_tools as a Maven plugin

I had a bad experience with the migration-tools. First, it was unable to handle declarations such as

<relativePath>../parent</relativePath>

(It expected a file, when directory was provided, it did not default to pom.xml in the directory, which is maven's behavior).

Next, the generation of workspace took a very long time, and it was redownloading the same artifacts multiple times from the remote repository. (I have a maven reactor build, with large number of modules.)

Finally, the generate_workspace.bzl contained dependencies on artifacts from inside the generated project. The way to get rid of that was to set a repository which does not have snapshot builds of my project. That would cause to skip these.

Given all that, I believe a good way to avoid most of the above is to offload the burden to Maven itself, and write a Maven plugin to perform this task.

The core of this functionality is essentially a rip-off of the dependency:tree plugin. I experimented with that and what I have is a yet another 80% solution, but I believe the Maven plugin idea has much to recommend.

Homebrew formula for generate_workspace

This is a feature request.

It would be great if a user on OS X could install generate_workspace without having to clone+build the repository.

I'm happy to do this work.

"bazel run //generate_workspace" regressed at HEAD (a5e68740)

previous to the current HEAD the following would run as expected:

bazel run //generate_workspace -- --artifact=io.dropwizard.metrics:metrics-core:3.1.2

producting the expected generate_workspace.bzl

however, as of a5e6874, the same command above prints the following warning and generates an empty bzl file:

WARNING: Could not find any repositories that knew how to resolve io.dropwizard.metrics:metrics-core:3.1.2 (checked )

As of the commit referenced above, there seems to be a new --repositories flag, that when empty yields the rather unhelpful warning above (i.e. "checked [blank]").

However when I try filling in a value for --repositories I see something like:

$ bazel run //generate_workspace -- --artifact=io.dropwizard.metrics:metrics-core:3.1.2 --repositories=https://mvnrepository.com/
....
WARNING: Unable to resolve version

again, creating an empty generate_workspace.bzl file.

Is this expected? If so can the documentation please be updated with new instructions.

Avoid dyndns.org lookup

dyndns.org is broken and doesn't give any response. We have a couple hundred dependencies and this the url connection won't stop until it times out after like 30 seconds. This results in that the entire dependency generation process takes longer than a couple hours.

We should avoid trying to download pom file from their site. Adding a simple blacklist can fix this issue.

#87

The tooling in it's current state is not realistic for Java Microservice codebases.

Bazel is fantastic and I am very happy with the characteristics of our Build. I am writing quite complex rules for our build now as we have specific packaging and launch requirements.

The tooling in this repo works fine for well defined workspaces with a few top level entries. I started our port to Bazel using the tooling here and quickly gave up.

To give you some scope -- at the moment I have partially ported our Java codebase to Bazel. I have 250 ~ external jars. There are a lot of dependencies which have huge transitive chains and conflicting common dependencies -- For example: Jersey, Spring-Data-JPA. Soon I need to update some of these dependencies and I can't see how I would realistically do that with the rules here.

To port our repo I took the approach of writing a shell script for the tooling here and then adding the dependencies incrementally. Each new dependency increased the cycle time. It was unbearable by the 10th or so dependency.

The transitive rule / macro provided is a non starter for a large project as well.

I ended up usingthese rules.

The following elements are key I think:

A dedicated dependency resolver: the rules above use Gradle (which uses Ivy I think).
Something serving as a lock file: The rules above spit out the transitive closure of the dependencies which have to be pasted back into the rules.

The approach isn't perfect as I still need to clutter my workspace with boilerplate, and I need to manually paste the transitive block in.

The Kubernetes project -- a go project uses the two elements above as well. The vendored dependencies are managed by dep and thus a lock file. And dep serves as a fast resolver.

It would be great to have a tool like the go dep tool, something that calculates a transitive dependency block and locks it and a file format that Bazel core understands.

transitive_maven_jar does not get triggered

Hi everyone,

I've been trying to get transitive_maven_jar rule to work, but fail to get it triggered at all.

I have run generate_workspace and produced stored the output in dependencies.bzl in the workspace root directory. I have confirmed that this works fine.

Now I've extend the WORKSPACE with the transitive_maven_jar rule, such as the following:

http_archive(
	name = "trans_maven_jar",
	url = "https://github.com/bazelbuild/migration-tooling/archive/master.zip",
	type = "zip",
	strip_prefix = "migration-tooling-master",
)

load("@trans_maven_jar//transitive_maven_jar:transitive_maven_jar.bzl", "transitive_maven_jar")


transitive_maven_jar(
    name = "dependencies",
    artifacts = [
                "org.apache.spark:spark-core_2.10:1.6.1",
                ...
    ]
)

load("//:dependencies.bzl", "generated_maven_jars")
generated_maven_jars()

However, when I run bazel build //some-package, it builds some-package and no signe of the transitive_maven_jar being evaluated.

I then update it with a new artifact:

transitive_maven_jar(
    name = "dependencies",
    artifacts = [
                "org.apache.spark:spark-core_2.10:1.6.1",
                "org.apache.spark:spark-mllib_2.10::2.2.2", # the new artifact
                ...
    ]
)

And still no sign of the transitive_maven_jar being evaluated during build time. I'm expecting to see a newly generated generate_workspace.bzl as the docs said, but nothing happens.

I'm new to bazel. Am I doing something wrong? How do I get the transitive_maven_jar to produce a new generated_workspace.bzl?

Thank you!

Fails with standard spring boot app

Hey,

it seems that the migration tool doesn't work with a spring boot app.

Just generate a standard spring boot app at https://start.spring.io/ and run the migration tooling on it.
I get the following error:

$ bazel run //generate_workspace -- --maven_project=/Users/chris/Desktop/demo/
INFO: Analysed target //generate_workspace:generate_workspace (0 packages loaded).
INFO: Found 1 target...
Target //generate_workspace:generate_workspace up-to-date:
  bazel-bin/generate_workspace/generate_workspace.jar
  bazel-bin/generate_workspace/generate_workspace
INFO: Elapsed time: 0.311s, Critical Path: 0.01s
INFO: Build completed successfully, 1 total action

INFO: Running command line: bazel-bin/generate_workspace/generate_workspace '--maven_project=/Users/chris/Desktop/demo/'
Jan 24, 2018 8:01:30 AM com.google.devtools.build.workspace.maven.DefaultModelResolver getRawModel
WARNUNG: Unable to resolve raw Maven model from /Users/chris/Desktop/demo/pom.xml: 1 problem was encountered while building the effective model for com.example:demo:0.0.1-SNAPSHOT
[FATAL] Non-resolvable parent POM: Could not find any repositories that knew how to resolve org.springframework.boot:spring-boot-starter-parent:1.5.9.RELEASE (checked ) @ 

Jan 24, 2018 8:01:30 AM com.google.devtools.build.workspace.maven.DefaultModelResolver getEffectiveModel
WARNUNG: Unable to resolve Maven model from /Users/chris/Desktop/demo/pom.xml: 1 problem was encountered while building the effective model for com.example:demo:0.0.1-SNAPSHOT
[FATAL] Non-resolvable parent POM: Could not find any repositories that knew how to resolve org.springframework.boot:spring-boot-starter-parent:1.5.9.RELEASE (checked ) @ 

Wrote /private/var/tmp/_bazel_chris/7cdadfef246225c2d3b62dd8a9e408e2/execroot/__main__/bazel-out/darwin-fastbuild/bin/generate_workspace/generate_workspace.runfiles/__main__/generate_workspace.bzl

Did i forgot any configuration or is it a bug?

May be related to: #61

Unpredictable "missing input file" error

Twice we have seen an error looking like this:

ERROR: missing input file '@org_apache_httpcomponents_httpclient//jar:httpclient-4.0.1.jar'
ERROR: [SNIP]/src/java/BUILD.bazel:105:1: //src/java:cloud_registry: missing input file '@org_apache_httpcomponents_httpclient//jar:httpclient-4.0.1.jar'
ERROR: [SNIP]/src/java/BUILD.bazel:105:1 1 input file(s) do not exist
INFO: Elapsed time: 420.080s, Critical Path: 127.06s
FAILED: Build did NOT complete successfully

The missing jar is not always the same. Retrying the build doesn't help, but bazel clean --expunge does. Unfortunately I don't have a way to reproduce it, but it seems to happen when the system hasn't rebuilt the Java targets in a while (maybe due to changes in the version on Maven?)

I know this isn't a very helpful bug report :) I'm posting in case anyone else is experiencing the same issue.

Allow specifying exclusions to generate_workspace

One issue my team is having with the existing generate_workspace tool (in the bazelbuild repo) is that we're generating dependencies for a maven artifact (google-cloud-logging) which includes some java netty dependencies, but in another place in the application we're including the pubfref protobuf rules for building java protobufs for grpc, and so are including netty deps from there via:

"@org_pubref_rules_protobuf//java:netty_runtime_deps"

So we're having an issue where we have two sets of netty dependencies being included of conflicting versions. Ideally I would like to tell generate_workspace that I have certain dependencies satisfied already, perhaps via a command line flag I can tell it that certain netty targets are satisfied externally through the pubref rules and it will generate a BUILD where the rules there depend on the pubref netty instead of duplicating the dependency.

I understand that we're using generate_workspace in an unintended way (we run it again each time that we add a new dependency, this is because the -b flag of the other tool is not working).

Is there a recommended solution for overlapping dependencies like this?

Thanks.

Transitive dependencies are not always included in java_library

Migrated from issue #2310 in main repository.

Description

When using generate_workspace transitive dependencies are not always included in the exports parameter of java_library. This does not always happen (one guess would be that this happens when two artifacts share some common dependencies).

Example

bazel run //src/tools/generate_workspace -- --artifact=com.twitter:twitter-server_2.11:1.25.0

Output:

java_library(
    name = "com_twitter_twitter_server_2_11",
    visibility = ["//visibility:public"],
    exports = [
        "@com_twitter_twitter_server_2_11//jar",
        "@com_fasterxml_jackson_core_jackson_annotations//jar",
        "@com_fasterxml_jackson_core_jackson_core//jar",
        "@com_fasterxml_jackson_core_jackson_databind//jar",
        "@com_fasterxml_jackson_module_jackson_module_paranamer//jar",
        "@com_fasterxml_jackson_module_jackson_module_scala_2_11//jar",
        "@com_github_ben_manes_caffeine_caffeine//jar",
        "@com_google_code_findbugs_jsr305//jar",
        "@com_google_guava_guava//jar",
        "@com_thoughtworks_paranamer_paranamer//jar",
        "@com_twitter_finagle_base_http_2_11//jar",
        "@com_twitter_finagle_core_2_11//jar",
        "@com_twitter_finagle_http_2_11//jar",
        "@com_twitter_finagle_netty4_2_11//jar",
        "@com_twitter_finagle_netty4_http_2_11//jar",
        "@com_twitter_finagle_thrift_2_11//jar",
        "@com_twitter_finagle_toggle_2_11//jar",
        "@com_twitter_finagle_zipkin_core_2_11//jar",
        "@com_twitter_libthrift//jar",
        "@com_twitter_scrooge_core_2_11//jar",
        "@com_twitter_util_app_2_11//jar",
        "@com_twitter_util_cache_2_11//jar",
        "@com_twitter_util_codec_2_11//jar",
        "@com_twitter_util_collection_2_11//jar",
        "@com_twitter_util_core_2_11//jar",
        "@com_twitter_util_events_2_11//jar",
        "@com_twitter_util_function_2_11//jar",
        "@com_twitter_util_hashing_2_11//jar",
        "@com_twitter_util_jvm_2_11//jar",
        "@com_twitter_util_lint_2_11//jar",
        "@com_twitter_util_logging_2_11//jar",
        "@com_twitter_util_registry_2_11//jar",
        "@com_twitter_util_stats_2_11//jar",
        "@commons_lang_commons_lang//jar",
        "@io_netty_netty//jar",
        "@io_netty_netty_buffer//jar",
        "@io_netty_netty_codec//jar",
        "@io_netty_netty_codec_http//jar",
        "@io_netty_netty_codec_socks//jar",
        "@io_netty_netty_common//jar",
        "@io_netty_netty_handler//jar",
        "@io_netty_netty_handler_proxy//jar",
        "@io_netty_netty_resolver//jar",
        "@io_netty_netty_transport//jar",
        "@javax_servlet_servlet_api//jar",
        "@org_scala_lang_modules_scala_parser_combinators_2_11//jar",
        "@org_scala_lang_scala_library//jar",
        "@org_scala_lang_scala_reflect//jar",
        "@org_slf4j_slf4j_api//jar",
    ],
)

bazel run //src/tools/generate_workspace -- \
  --artifact=com.twitter:finagle-thrift_2.11:6.40.0 \
  --artifact=com.twitter:twitter-server_2.11:1.25.0

Output (transitive dependencies are missing):

java_library(
    name = "com_twitter_twitter_server_2_11",
    visibility = ["//visibility:public"],
    exports = [
        "@com_twitter_twitter_server_2_11//jar",
        "@com_fasterxml_jackson_core_jackson_core//jar",
        "@com_fasterxml_jackson_core_jackson_databind//jar",
        "@com_fasterxml_jackson_module_jackson_module_scala_2_11//jar",
        "@com_google_guava_guava//jar",
        "@com_twitter_finagle_base_http_2_11//jar",
        "@com_twitter_finagle_core_2_11//jar",
        "@com_twitter_finagle_http_2_11//jar",
        "@com_twitter_finagle_netty4_2_11//jar",
        "@com_twitter_finagle_netty4_http_2_11//jar",
        "@com_twitter_finagle_thrift_2_11//jar",
        "@com_twitter_finagle_toggle_2_11//jar",
        "@com_twitter_finagle_zipkin_core_2_11//jar",
        "@com_twitter_libthrift//jar",
        "@com_twitter_scrooge_core_2_11//jar",
        "@com_twitter_util_app_2_11//jar",
        "@com_twitter_util_codec_2_11//jar",
        "@com_twitter_util_collection_2_11//jar",
        "@com_twitter_util_core_2_11//jar",
        "@com_twitter_util_events_2_11//jar",
        "@com_twitter_util_jvm_2_11//jar",
        "@com_twitter_util_lint_2_11//jar",
        "@com_twitter_util_logging_2_11//jar",
        "@com_twitter_util_registry_2_11//jar",
        "@com_twitter_util_stats_2_11//jar",
        "@commons_lang_commons_lang//jar",
        "@io_netty_netty_codec_http//jar",
        "@org_scala_lang_scala_library//jar",
    ],
)

Potential causes may be found here: https://github.com/bazelbuild/bazel/blob/master/src/tools/generate_workspace/src/main/java/com/google/devtools/build/workspace/maven/Resolver.java#L166

New instance of Rule is created and transitive dependencies are resolved and added only in the case that the dependency is new. If the dependency already exists the dependencies will never be populated and https://github.com/bazelbuild/bazel/blob/master/src/tools/generate_workspace/src/main/java/com/google/devtools/build/workspace/maven/Resolver.java#L185 will have no effect.

Zip Release

Thanks for all your work on this! I'm wondering if we could get a tagged release; specifically I'd like to include the project via bazel's http_archive rule, but I need a released zip file for this. Thanks!

Aether dependency resolution fails with "Invalid Range Result", generates partial workspace

As the title suggests, the generate_workspace tool after PR #45 ends up omitting some dependencies due to a problem with the new dependency resolution mechanism.

I noticed this problem when one of my dependencies attempted to pull in com.google.inject:guice:4.0 but the resulting generate_workspace.bzl did not contain corresponding maven_jar/java_library rules.

I've created a minimal test project: https://github.com/Zetten/bazel-generate-workspace-error

The pom.xml file describes Guava and Guice dependencies, but if the generate_workspace command is executed against it (may be run easily from the project dir with bazel build :test_generate_workspace) the resulting .bzl file does not contain the Guice dependency (the Guava deps are fine). The build log reports:

Aug 02, 2017 6:55:33 PM com.google.devtools.build.workspace.maven.Resolver traverseDeps
INFO: 	Downloading pom for com.example:bazel-generate-error:0.0.0
Aug 02, 2017 6:55:34 PM com.google.devtools.build.workspace.maven.Resolver traverseDeps
INFO: 	Downloading pom for com.google.guava:guava:22.0
Aug 02, 2017 6:55:34 PM com.google.devtools.build.workspace.maven.Resolver traverseDeps
INFO: 	Downloading pom for com.google.code.findbugs:jsr305:2.0.2
Aug 02, 2017 6:55:34 PM com.google.devtools.build.workspace.maven.Resolver traverseDeps
INFO: 	Downloading pom for com.google.errorprone:error_prone_annotations:2.0.18
Aug 02, 2017 6:55:35 PM com.google.devtools.build.workspace.maven.Resolver traverseDeps
INFO: 	Downloading pom for com.google.j2objc:j2objc-annotations:1.1
Aug 02, 2017 6:55:35 PM com.google.devtools.build.workspace.maven.Resolver traverseDeps
INFO: 	Downloading pom for org.codehaus.mojo:animal-sniffer-annotations:1.14
Aug 02, 2017 6:55:35 PM com.google.devtools.build.workspace.maven.Resolver addDependency
WARNING: Could not resolve dependency com.google.inject:guice:4.0: Unable to find a version for com.google.inject:guice:4.0 due to Invalid Range Result

${project.basedir} not interpolated in repositories

This bug may just be "interpolation does not happen anywhere", but I'm specifically running into issues with local repositories like this:

<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                      https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <repositories>
    <repository>
      <id>project.local</id>
      <name>project</name>
      <url>file:${project.basedir}/repo</url>
    </repository>
  </repositories>

  <groupId>com.example</groupId>
  <artifactId>example</artifactId>
  <version>0.1.0-SNAPSHOT</version>
  <packaging>pom</packaging>
</project>

The repo directory here would contain a local maven repository with artifacts.

For whatever reason, the resolver is never getting the ${project.basedir} value interpolated, and therefore sees a bad filename.

I've tried numerous changes, but can't figure out what's missing. The relevant section of the docs implies it should Just Happen, although I also think that the wiping of profiles that the model resolver does is contributing the problem. Without any profiles set, this won't get interpolated - although removing that line doesn't actually fix the problem.

\VCVARSALL.BAT doesn't exist, please check your VC++ installation

D:\workspace\netroby\migration-tooling [master ≡]
λ  bazel run //generate_workspace
......................
DEBUG: C:/users/huzhifeng/appdata/local/temp/_bazel_huzhifeng/g1u2avra/external/bazel_tools/tools/cpp/lib_cc_configure.bzl:37:3:
Auto-Configuration Warning: 'BAZEL_VC' is not set, start looking for the latest Visual C++ installed.
DEBUG: C:/users/huzhifeng/appdata/local/temp/_bazel_huzhifeng/g1u2avra/external/bazel_tools/tools/cpp/lib_cc_configure.bzl:37:3:
Auto-Configuration Warning: Looking for VS%VERSION%COMNTOOLS environment variables,eg. VS140COMNTOOLS
DEBUG: C:/users/huzhifeng/appdata/local/temp/_bazel_huzhifeng/g1u2avra/external/bazel_tools/tools/cpp/lib_cc_configure.bzl:37:3:
Auto-Configuration Warning: Looking for Visual C++ through registry
DEBUG: C:/users/huzhifeng/appdata/local/temp/_bazel_huzhifeng/g1u2avra/external/bazel_tools/tools/cpp/lib_cc_configure.bzl:37:3:
Auto-Configuration Warning: Visual C++ build tools found at D:\soft\Microsoft Visual Studio\2017\Community\\VC
ERROR: in target '//external:cc_toolchain': no such package '@local_config_cc//': Traceback (most recent call last):
        File "C:/users/huzhifeng/appdata/local/temp/_bazel_huzhifeng/g1u2avra/external/bazel_tools/tools/cpp/cc_configure.bzl", line 37
                configure_windows_toolchain(repository_ctx)
        File "C:/users/huzhifeng/appdata/local/temp/_bazel_huzhifeng/g1u2avra/external/bazel_tools/tools/cpp/windows_cc_configure.bzl", line 323, in configure_windows_toolchain
                _find_env_vars(repository_ctx, vc_path)
        File "C:/users/huzhifeng/appdata/local/temp/_bazel_huzhifeng/g1u2avra/external/bazel_tools/tools/cpp/windows_cc_configure.bzl", line 194, in _find_env_vars
                _find_vcvarsall_bat_script(repository_ctx, vc_path)
        File "C:/users/huzhifeng/appdata/local/temp/_bazel_huzhifeng/g1u2avra/external/bazel_tools/tools/cpp/windows_cc_configure.bzl", line 188, in _find_vcvarsall_bat_script
                auto_configure_fail((vcvarsall + " doesn't exist, pl..."))
        File "C:/users/huzhifeng/appdata/local/temp/_bazel_huzhifeng/g1u2avra/external/bazel_tools/tools/cpp/lib_cc_configure.bzl", line 30, in auto_configure_fail
                fail(("\n%sAuto-Configuration Error:%...)))

Auto-Configuration Error: D:\soft\Microsoft Visual Studio\2017\Community\\VC\Auxiliary\Build\VCVARSALL.BAT doesn't exist, please check your VC++ installation
INFO: Elapsed time: 5.008s
FAILED: Build did NOT complete successfully (2 packages loaded)
ERROR: Build failed. Not running target

Multiple Bazel Project Workspace Generator

The previous version of the tool appeared to allow one to specify various Bazel projects and a combined WORKSPACE file would be generated. Is that something that we might be able to see with this project as well?

Clarify bazelbuild/migration-tooling VS johnynek/bazel-deps

https://docs.bazel.build/versions/master/generate-workspace.html points to this repo.

But the code in this repo appears to be broken, and seems to have been in a while.

There are folks opening issues in this repo and it doesn't look like anyone is managing it?

#80 (for #47) says "we've switched to https://github.com/johnynek/bazel-deps instead".

Should this repo be removed and the documentation updated?

Or should more people be made committers in this repo?

Support dependencyManagement sections within Maven

Migrating from issue 307 in main repository

See https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/bazel-dev/CuGGIbMnuYM/fGzfSY1CjOsJ

generate_workspace may pull in too much from dependencyManagement section of POMs

Running bazel run //generate_workspace -- '--artifact=org.apache.beam:beam-sdks-java-core:2.0.0' took >1hr and generated a >100k bzl file.

If I remove the JARs pulled in via the parent POMs dependencyManagement, the result appears reasonable.

Should generate_workspace pull in every dependency specified in the full set of parent POMs or just use their versions to override the child POMs versions? Or should only a certain set of JARs brought in through dependencyManagement be used?

Check for .class file overlaps

Migrating from issue 358 in main repository.

If two jars contain the same .class files, it would be nice for generate_workspace to warn the user. Probably put this behind a flag.

generate_workspace is generating invalid shas

Migrated from issue #1131 in original repository.

generate_workspace seems to be creating invalid sha1 entries of the form:

maven_jar(
    name = "net_java_dev_jets3t_jets3t",
    artifact = "net.java.dev.jets3t:jets3t:0.6.1",
    sha1 = "9f95b944ccbbbeedd397e8f62e5690fd735a45e5  maven2/net/java/dev/jets3t/jets3t/0.6.1/jets3t-0.6.1.jar",
)

I'm currently synced to 653869e (HEAD~1 because HEAD seems to be broken).

It's including the rest of the .sha1 file it downloaded, not cutting it off after the sha. See https://groups.google.com/d/msg/bazel-discuss/gu8RbSdD0xY/YvI3f9azAQAJ.

One potential cause: https://github.com/bazelbuild/bazel/blob/master/src/tools/generate_workspace/src/main/java/com/google/devtools/build/workspace/maven/Resolver.java#L275?

generate_workspace does not work with spark artifact

This issue is migrated from issue 2536 of bazel repository.

A minimal example to reproduce the problem:

bazel run //generate_workspace -- --artifact=org.apache.spark:spark-core_2.11:2.1.0

Description of the problem / feature request / question:

INFO: Running command line: bazel-bin/generate_workspace/generate_workspace '--artifact=org.apache.spark:spark-core_2.11:2.1.0'
Aug 31, 2017 3:55:51 PM com.google.devtools.build.workspace.maven.DefaultModelResolver getEffectiveModel
WARNING: Unable to resolve Maven model from https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.11/2.1.0/spark-core_2.11-2.1.0.pom: 1 problem was encountered while building the effective model for org.apache.spark:spark-core_2.11:2.1.0
[ERROR] 'dependencyManagement.dependencies.dependency.groupId' for ${jline.groupid}:jline:jar with value '${jline.groupid}' does not match a valid id pattern. @

Parent project with no artifacts fails

Having a dependency on a parent project with only a POM file fails due to the fact that parent projects are resolved using requestVersionList.

This project will fail with the error [FATAL] Non-resolvable parent POM org.sonatype.oss:oss-parent:9: Unable to resolve version @.

<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                      https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <parent>
    <groupId>org.sonatype.oss</groupId>
    <artifactId>oss-parent</artifactId>
    <version>9</version>
  </parent>

  <groupId>com.example</groupId>
  <artifactId>example</artifactId>
  <version>0.1.0-SNAPSHOT</version>
  <packaging>pom</packaging>
</project>

I can't find an Aether method to download only a POM file. This might need to be handled outside of regular dependency resolution, say by looping through repository URLs until a valid one is found.

Unable to resolve Maven Model with missing dependencies.dependency.verison

Any idea what this issue is? The Artifact is on maven repo. Happening with several other artifacts as well

WARNING: Unable to resolve Maven model from https://repo1.maven.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-xml/2.8.8/jackson-dataformat-xml-2.8.8.pom: 2 problems were encountered while building the effective model for com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.8.8
[ERROR] 'dependencies.dependency.version' for com.fasterxml.jackson.core:jackson-annotations:jar is missing. @
[ERROR] 'dependencies.dependency.version' for junit:junit:jar is missing. @

New project Bazel Java Dependency Converter

Bazel force people to specify dependencies explicitly. This is good to spot some version conflicts but a heavy burden without tool supporting.

The official migrating tool for Maven is deprecated, while its alternative bazel-deps didn’t support Maven project smoothly. That’s why I need to write this project.

https://github.com/hcoona/bazel-java-dependency-converter

generate_workspace does not create automatically output directory if they do not exist

$ bazel run //generate_workspace -- -a com.lesfurets:jenkins-pipeline-unit:1.0 -o /tmp/test
INFO: Analysed target //generate_workspace:generate_workspace.
INFO: Found 1 target...
Target //generate_workspace:generate_workspace up-to-date:
  bazel-bin/generate_workspace/generate_workspace.jar
  bazel-bin/generate_workspace/generate_workspace
INFO: Elapsed time: 0.277s, Critical Path: 0.02s
INFO: Build completed successfully, 1 total action

INFO: Running command line: bazel-bin/generate_workspace/generate_workspace -a com.lesfurets:jenkins-pipeline-unit:1.0 -o /tmp/test
Jun 28, 2017 6:34:20 PM com.google.devtools.build.workspace.maven.Resolver traverseDeps
INFO: 	Downloading pom for com.lesfurets:jenkins-pipeline-unit:1.0
Jun 28, 2017 6:34:21 PM com.google.devtools.build.workspace.maven.Resolver traverseDeps
INFO: 	Downloading pom for org.codehaus.groovy:groovy-all:2.4.6
Jun 28, 2017 6:34:23 PM com.google.devtools.build.workspace.maven.Resolver traverseDeps
INFO: 	Downloading pom for com.cloudbees:groovy-cps:1.12
Jun 28, 2017 6:34:25 PM com.google.devtools.build.workspace.maven.Resolver traverseDeps
INFO: 	Downloading pom for com.google.guava:guava:11.0.1
Jun 28, 2017 6:34:26 PM com.google.devtools.build.workspace.maven.Resolver traverseDeps
INFO: 	Downloading pom for com.google.code.findbugs:jsr305:1.3.9
Jun 28, 2017 6:34:29 PM com.google.devtools.build.workspace.maven.Resolver traverseDeps
INFO: 	Downloading pom for commons-io:commons-io:2.5
Jun 28, 2017 6:34:33 PM com.google.devtools.build.workspace.maven.Resolver traverseDeps
INFO: 	Downloading pom for org.apache.ivy:ivy:2.4.0
Jun 28, 2017 6:34:33 PM com.google.devtools.build.workspace.output.BzlWriter write
SEVERE: Could not write /tmp/test/generate_workspace.bzl: /tmp/test/generate_workspace.bzl (No such file or directory)

Add support for artifactory and private repositories

Migrating from issue 498 in main repository.

This is already implemented on the Bazel side, so similar code can probably be used for generate_workspace. It should use all of the local (settings.xml) configuration options.

An example from the original post:

I think I've run into a similar requirement; I'm trying to use robolectric as in:

bazel run //src/tools/generate_workspace -- --artifact=org.robolectric:robolectric:3.1.2

However, robolectric depends on maven-ant-tasks which depends on some org.codehaus.plexus artifacts. Codehaus.org no longer appears to host code, but bazel is attempting to use it to retrieve the plexus assets:

ERROR: Unable to resolve Maven model from http://snapshots.maven.codehaus.org/maven2/org/codehaus/plexus/plexus-utils/1.0.4/plexus-utils-1.0.4.pom: 1 problem was encountered while building the effective model
[FATAL] Non-parseable POM http://snapshots.maven.codehaus.org/maven2/org/codehaus/plexus/plexus-utils/1.0.4/plexus-utils-1.0.4.pom: Expected root element 'project' but found 'html' (position: START_TAG seen \n... @2:7) @ line 2, column 7

I think that if I could specify a settings.xml, then I might be able to resolve this with http://www.codehaus.org/mechanics/maven/

or:

http://stackoverflow.com/questions/30288097/how-to-resolve-codehaus-service-unavailable-maven-build-error

But since this issue seems to state that there is no way to specify a settings.xml when using generate_workspace, does anyone know of some workaround I can use in the meantime?

Thanks!

generate_workspace does not pickup http_proxy or https_proxy settings

Migrating from issue 1800 in bazel repository.

Thus, it fails to run/fetch anything if you are behind a proxy. Optionally or alternatively, it could use .m2/settings.xml's proxy settings to configure proxies if that's easier.

generate_workspace doesn't include runtime dependencies of maven artifact

Migrated from issue 2316 in main repository

example:
bazel run //src/tools/generate_workspace -- --artifact=org.mockito:mockito-core:2.4.2

mockito-core has 4 runtime dependencies, none were generated.

I believe the cause of this issue is this line in the Maven Resolver:

public Model resolveEffectiveModel(ModelSource modelSource, Set<String> exclusions, Rule parent) {
    Model model = modelResolver.getEffectiveModel(modelSource, handler);
    ...
    for (Dependency dependency : model.getDependencies()) {
      if (!dependency.getScope().equals(COMPILE_SCOPE)) {
        continue;
      }
    ...
    }

Any maven dependencies that aren't scoped for compile time are ignored.

Provide tooling to import multiple versions of an artifact

Migrated from issue #1783 in main repository.

E.g., generate_workspace --artifact=jetty:1,old_prod --artifact=jetty:2,prod will create two maven_jars with the given suffixes and their dependencies, matching deps where possible and appending the suffix where not, e.g.,

jetty_prod -> foo_prod -> bar
jetty_old_prod -> foo_old_prod -> bar

jetty, foo, and bar are maven artifacts, dep on bar is the same version for both, dep on foo is not.

Make generate_workspace visible

Can we set visibility to public on the generate_workspace? I'd love to be able to automatically pull the tool into some of my bazel projects so I can use it more easily, but the target is currently private.

Could not resolve dependency due to Invalid Range Result

I'm fresh to bazel and encounter some issue when generating workspace from a simple maven project, I don't know why it Could not resolve dependency ... due to Invalid Range Result on every dependencies, could anyone help?

➜  migration-tooling-master bazel run //generate_workspace -- -m /home/novemser/Documents/Code/Java/                             
INFO: Analysed target //generate_workspace:generate_workspace (0 packages loaded).
INFO: Found 1 target...
Target //generate_workspace:generate_workspace up-to-date:
  bazel-bin/generate_workspace/generate_workspace.jar
  bazel-bin/generate_workspace/generate_workspace
INFO: Elapsed time: 0.177s, Critical Path: 0.00s
INFO: Build completed successfully, 1 total action

INFO: Running command line: bazel-bin/generate_workspace/generate_workspace -m /home/novemser/Documents/Code/Java/
Mar 09, 2018 12:41:52 PM com.google.devtools.build.workspace.maven.Resolver traverseDeps
INFO: 	Downloading pom for com.novemser:learnJava:1.0-SNAPSHOT
Mar 09, 2018 12:41:52 PM com.google.devtools.build.workspace.maven.Resolver addDependency
WARNING: Could not resolve dependency org.apache.spark:spark-core_2.11:2.3.0: Unable to find a version for org.apache.spark:spark-core_2.11:2.3.0 due to Invalid Range Result
Mar 09, 2018 12:41:52 PM com.google.devtools.build.workspace.maven.Resolver addDependency
WARNING: Could not resolve dependency junit:junit:4.12: Unable to find a version for junit:junit:4.12 due to Invalid Range Result
Wrote /home/novemser/.cache/bazel/_bazel_novemser/2f686a976071dd49bc361f578d3f8c1e/execroot/__main__/bazel-out/k8-fastbuild/bin/generate_workspace/generate_workspace.runfiles/__main__/generate_workspace.bzl

My pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.novemser</groupId>
    <artifactId>learnJava</artifactId>
    <version>1.0-SNAPSHOT</version>
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
    </properties>

    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.3.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/junit/junit -->
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
</project>

How to add a new maven deps and upgrade old maven deps to WORKSPACE file

Migrated from issue 989 in main repository

Post 1

I see the generate WORKSPACE tool, which is useful.

But an ongoing issue is the pain in transitively adding the N^th dependency. Many of the transitive deps will already have been added, or added with different versions.

It is not clear the best way to tackle this problem.

Ideally, we could run a tool that would load the WORKSPACE, it would merge in a new POM and output a new updated WORKSPACE and BUILD. We would want to control what do to when we see redundant artifacts. Some ideas:

force the user to manually resolve.
just try the latest version and test.
use the tests to try to automatically pick a version (prefer later, but verify by running a test command).
set up a second, differently named external repo pointing at the jar. Find a way to error or warn if two versions of the same artifact wind up in the transitive deps of any one target.

Solution 4. might be ideal since actually putting exactly one version of every target might not be workable.

Post 2

a related issue is upgrading deps. Many will need to be upgraded at a time. If we are using hashes in maven_jar those will all need to be updated and that could be a pretty tedious job by hand.

Post 3

by the way, we have pretty much solved this problem with a tool we run along side bazel:
https://github.com/johnynek/bazel-deps

we maintain our maven dependencies in a yaml file, then it generates the transitive deps with shas and all the targets for the maven graph, normalized to one version per dependency.

It would be cool to have a repository rule do this from within bazel, but there is currently no way great way to do that which I can see, since we can't run jvm code from inside a repository rule. I guess we could pull a compiled jar, then run that jar to generate the dependencies, but so far our dependencies change rarely enough that an external tool is fine.

Outputting currently resolved artifact to console

This is a simple aesthetic change. It would be nice if generate_workspace could output what artifact it is currently resolving. For example:

INFO: Running command line: //generate_workspace '--maven_project=/helloworld' '--artifact=org.powermock:powermock-module-junit4:1.5.1' '--artifact=org.powermock:powermock-api-mockito:1.5.1'
    ...
    Resolving model org.powermock powermock-module-junit4 1.5.1
    Resolving model org.powermock powermock-modules 1.5.1
    Resolving model org.powermock powermock 1.5.1
    Resolving model org.sonatype.oss oss-parent 5
    Resolving model junit junit 4.11
    Resolving model org.hamcrest hamcrest-core 1.3
    Resolving model org.hamcrest hamcrest-parent 1.3
    Resolving model org.powermock powermock-module-junit4-common 1.5.1
    Resolving model org.powermock powermock-modules 1.5.1
    Resolving model org.powermock powermock 1.5.1
    Resolving model org.sonatype.oss oss-parent 5

@kchodorow, how would you like it to output the currently resolved artifact? In addition, when there are duplicates, do you want it to skip over those duplicates?

About the time to build the transitive_maven_jar

Hi, I have tried the transitive_maven_jar. I find that the cost of re-fetching dependencies everytime is very heavy. It cost a lot of time. Are there any solution to speed up it?

Generated libraries from maven should export transitive dependencies

The generate_workspace tool generates both maven_jar targets and also java_library targets.

The java_library targets seem to contain transitive dependencies, for example, when generating dependencies for the org.asynchttpclient:async-http-client:2.5.2 artifact, I see that a java_library target is created that looks like this:

  native.java_library(
      name = "org_asynchttpclient_async_http_client",
      visibility = ["//visibility:public"],
      exports = ["@org_asynchttpclient_async_http_client//jar"],
      runtime_deps = [
          ":com_sun_activation_javax_activation",
          ":com_typesafe_netty_netty_reactive_streams",
          ":io_netty_netty_buffer",
          ":io_netty_netty_codec",
          ":io_netty_netty_codec_dns",
          ":io_netty_netty_codec_http",
          ":io_netty_netty_codec_socks",
          ":io_netty_netty_common",
          ":io_netty_netty_handler",
          ":io_netty_netty_handler_proxy",
          ":io_netty_netty_resolver",
          ":io_netty_netty_resolver_dns",
          ":io_netty_netty_transport",
          ":io_netty_netty_transport_native_epoll",
          ":io_netty_netty_transport_native_unix_common",
          ":org_asynchttpclient_async_http_client_netty_utils",
          ":org_reactivestreams_reactive_streams",
          ":org_slf4j_slf4j_api",
      ],
  )

At first glance, it seems very handy to have these generated java_library targets containing all the transitive dependencies of a given maven jar. However, given that these are declared as runtime_deps, the generated library is not very useful because I can't use it as a single dependency in other targets. I still need to include all the transitive dependencies manually. So I don't understand the point of generated java_libray targets as maven_jars could be used directly.

One solution to this problem would be to make the java_library targets export the transitive dependencies. For example like this:

  native.java_library(
      name = "org_asynchttpclient_async_http_client",
      visibility = ["//visibility:public"],
      exports = [
          "@org_asynchttpclient_async_http_client//jar"
          "@com_sun_activation_javax_activation//jar",
          "@com_typesafe_netty_netty_reactive_streams//jar",
          "@io_netty_netty_buffer//jar",
          "@io_netty_netty_codec//jar",
          "@io_netty_netty_codec_dns//jar",
          "@io_netty_netty_codec_http//jar",
          "@io_netty_netty_codec_socks//jar",
          "@io_netty_netty_common//jar",
          "@io_netty_netty_handler//jar",
          "@io_netty_netty_handler_proxy//jar",
          "@io_netty_netty_resolver//jar",
          "@io_netty_netty_resolver_dns//jar",
          "@io_netty_netty_transport//jar",
          "@io_netty_netty_transport_native_epoll//jar",
          "@io_netty_netty_transport_native_unix_common//jar",
          "@org_asynchttpclient_async_http_client_netty_utils//jar",
          "@org_reactivestreams_reactive_streams//jar",
          "@org_slf4j_slf4j_api//jar",
      ],
  )

In this way, I could easily declare org_asynchttpclient_async_http_client as my dep and make sure all it's transitive dependencies will come with it.

Note: I'm new to Bazel. I apologize if this doesn't make sense :)

Aether dependency resolution fails and generates partial workspace

I'm not sure if this is the same bug as #47 with a different manifestation:

% bazel run //generate_workspace -- --output_dir /tmp --artifact=ru.vyarus:dropwizard-guicey:jar:4.0.1
INFO: Found 1 target...
Target //generate_workspace:generate_workspace up-to-date:
  bazel-bin/generate_workspace/generate_workspace.jar
  bazel-bin/generate_workspace/generate_workspace
INFO: Elapsed time: 0.072s, Critical Path: 0.00s

INFO: Running command line: bazel-bin/generate_workspace/generate_workspace --output_dir /tmp '--artifact=ru.vyarus:dropwizard-guicey:jar:4.0.1'
Aug 08, 2017 8:18:18 PM com.google.devtools.build.workspace.maven.DefaultModelResolver getEffectiveModel
WARNING: Unable to resolve Maven model from https://repo1.maven.org/maven2/ru/vyarus/dropwizard-guicey/4.0.1/dropwizard-guicey-4.0.1.pom: 4 problems were encountered while building the effective model for ru.vyarus:dropwizard-guicey:4.0.1
[FATAL] Non-resolvable parent POM com.google.inject:guice-parent:4.1.0: Unable to resolve version @ com.google.inject:guice-bom:[unknown-version]
[ERROR] 'dependencies.dependency.version' for com.google.inject:guice:jar is missing. @ 
[ERROR] 'dependencies.dependency.version' for com.google.inject.extensions:guice-servlet:jar is missing. @ 
[ERROR] 'dependencies.dependency.version' for com.google.inject.extensions:guice-multibindings:jar is missing. @ 

Wrote /tmp/generate_workspace.bzl

The generated file is bare:

# The following dependencies were calculated from:
#
# generate_workspace --output_dir /tmp --artifact=ru.vyarus:dropwizard-guicey:jar:4.0.1


def generated_maven_jars():
  native.maven_jar(
      name = "ru_vyarus_dropwizard_guicey",
      artifact = "ru.vyarus:dropwizard-guicey:4.0.1",
  )




def generated_java_libraries():
  native.java_library(
      name = "ru_vyarus_dropwizard_guicey",
      visibility = ["//visibility:public"],
      exports = ["@ru_vyarus_dropwizard_guicey//jar"],
  )

If I check out an old commit before #45, the generated file is full of dependencies. I also tried the fix in #50, but it doesn't help this scenario.

[Upgrading] Errors using generate_workspace to add a new external dependency

Context

Building off of the stable changes made in #29, I tried adding a dependency on maven-settings:3.5.0, so that I could use a org.apache.maven.settings.building.SettingsBuilder within one of my classes. However, I was unsuccessful in building the target with the new dependencies. I am unsure what I am doing incorrectly or forgetting to do.

Here is the error reported when I ran bazel build generate_workspace/src/main/java/com/google/devtools/build/workspace/maven:all

migration-tooling/generate_workspace/src/main/java/com/google/devtools/build/workspace/maven/BUILD:1:1: Java compilation in rule '//generate_workspace/src/main/java/com/google/devtools/build/workspace/maven:maven' failed: Worker process sent response with exit code: 1.
generate_workspace/src/main/java/com/google/devtools/build/workspace/maven/DefaultModelResolver.java:44: error: package org.apache.maven.settings.building does not exist
import org.apache.maven.settings.building.DefaultSettingsBuildingRequest;
                                         ^
generate_workspace/src/main/java/com/google/devtools/build/workspace/maven/DefaultModelResolver.java:101: error: cannot find symbol
    DefaultSettingsBuildingRequest request = new DefaultSettingsBuildingRequest();
    ^
  symbol:   class DefaultSettingsBuildingRequest
  location: class DefaultModelResolver
generate_workspace/src/main/java/com/google/devtools/build/workspace/maven/DefaultModelResolver.java:101: error: cannot find symbol
    DefaultSettingsBuildingRequest request = new DefaultSettingsBuildingRequest();
                                                 ^
  symbol:   class DefaultSettingsBuildingRequest
  location: class DefaultModelResolver
Target //generate_workspace/src/main/java/com/google/devtools/build/workspace/maven:maven failed to build
Use --verbose_failures to see the command lines of failed build steps.

Reproducing

I went the route of running the generate_workspace command to modifying the workspace.bzl file, and then depending directly on the new artifacts maven jar.

To modify the workspace.bzl file, I ran the following command:

java -jar generate_workspace_deploy.jar \
	-a=com.google.code.findbugs:jsr305:3.0.1 \
	-a=org.codehaus.plexus:plexus-utils:jar:3.0.24 \
	-a=org.codehaus.plexus:plexus-component-annotations:1.7.1 \
	-a=org.codehaus.plexus:plexus-interpolation:1.24 \
	-a=com.google.guava:guava:20.0 \
	-a=org.apache.maven:maven-artifact:3.5.0 \
	-a=org.mockito:mockito-all:1.9.5 \
	-a=junit:junit:4.4 \
	-a=com.google.truth:truth:0.30 \
	-a=org.apache.maven:maven-settings:3.5.0 #<-- new addition

I used the stable version of generate_workspace from #29 to construct the deploy jar. This resulted in the following changes to the generate_workspace.bzl file

Then, I added deps on"@org_apache_maven_maven_settings//jar", in the "//generate_workspace/src/main/java/com/google/devtools/build/workspace/maven:maven target [diff].

After which point, I simply imported org.apache.maven.settings.building.DefaultSettingsBuildingRequest in the DefaultModelResolver [diff].

@kchodorow

generate_workspace should be able to use a configurable Maven Central URL

Migrated from issue 1782 in main repository

Either:

Picking it up from settings.xml
Give it a path to settings.xml
Or set a flag for Maven Central URL.