Documentation on testing toolchain changes is out of date about java_tools HOT 9 OPEN

For posterity, I think the complete steps for java_tools are something like:

bazel build //src:java_tools_zip
mkdir /tmp/java_tools/
unzip -q -o -d /tmp/java_tools/ bazel-bin/src/java_tools.zip
touch /tmp/java_tools/WORKSPACE

bazel build --override_repository=remote_java_tools=/tmp/java_tools ...

And for java_tools_linux:

bazel build //src:java_tools_prebuilt_zip
mkdir /tmp/java_tools_linux/
unzip -q -o -d /tmp/java_tools_linux/ bazel-bin/src/java_tools_prebuilt.zip
touch /tmp/java_tools_linux/WORKSPACE

--override_repository=rules_java~7.3.2~toolchains~remote_java_tools_linux=/tmp/java_tools_linux

I think bzlmod changes mean it's necessary to override version (e.g. the ~7.3.2~ in the name), maybe there's a cleaner alternative for that.

from java_tools.

Is this the right idea? I can't get it to work:

bazel build //src:java_tools_zip //src:java_tools_prebuilt_zip
unzip -q -o -d /tmp bazel-bin/src/java_tools.zip
unzip -q -o -d /tmp/java_tools_prebuilt bazel-bin/src/java_tools_prebuilt.zip
bazel build --override_repository=java_tools=/tmp/java_tools --override_repository=java_tools_prebuilt=/tmp/java_tools_prebuilt //src:bazel

I'm testing a repro for bazelbuild/bazel#14093, which causes JavaBuilder to unconditionally OOM, and the only way I can see the behaviour I'm introducing is by patching distdir_deps.bzl

from java_tools.

OK, there was 2 issues :

1. WORKSPACE file is missing from zip file generate from //src:java_tools_zip (you can copy over manually the default one)
2. --override_repository=java_tools=/tmp/java_tools should be --override_repository=remote_java_tools=/tmp/java_tools

After that, I can verify the files located at /tmp/java_tools are copied over to /external/remote_java_tools/java_tools and are used.

Seeing this error now (which can be temporary worked around by removing ErrorPronePlugin manually) :

Exception in thread "main" java.lang.NoSuchMethodError: 'com.google.common.collect.ImmutableMap com.google.common.collect.ImmutableMap$Builder.buildOrThrow()'
	at com.google.errorprone.scanner.ScannerSupplier.defaultSeverities(ScannerSupplier.java:64)
	at com.google.errorprone.scanner.ScannerSupplier.fromBugCheckerInfos(ScannerSupplier.java:85)
	at com.google.errorprone.scanner.BuiltInCheckerSuppliers.allChecks(BuiltInCheckerSuppliers.java:566)
	at com.google.errorprone.scanner.BuiltInCheckerSuppliers.errorChecks(BuiltInCheckerSuppliers.java:583)
	at com.google.devtools.build.buildjar.javac.plugins.errorprone.ErrorPronePlugin.<init>(ErrorPronePlugin.java:56)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.parse(BazelJavaBuilder.java:126)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.parseAndBuild(BazelJavaBuilder.java:76)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:65)

from java_tools.

I think the current process is something like the following. @comius does that look right, or is there a better way to do this?

• From the main Bazel repo, build //src:java_tools_zip and stash it somewhere:

  $ bazel build //src:java_tools_zip
  $ cp bazel-bin/src/java_tools.zip /tmp/
  

• Patch distdir_deps.bzl to refer to the local java_tools_zip:

  --- a/distdir_deps.bzl
  +++ b/distdir_deps.bzl
  @@ -267,11 +267,10 @@ DIST_DEPS = {
               "remote_java_tools_test",
               "remote_java_tools_for_testing",
           ],
  -        "archive": "java_tools-v11.3.zip",
  -        "sha256": "52b66d8df456f2ce057d8e435904789463df8414af8110aa13af68ce32d8c4cc",
  +        "archive": "java_tools3.zip",
  +        "sha256": "6e59059c571571528d7e5e264d6ec6bba4ba570241f71fc6d31f17cbbcb30fc8",
           "urls": [
  -            "https://mirror.bazel.build/bazel_java_tools/releases/java/v11.3/java_tools-v11.3.zip",
  -            "https://github.com/bazelbuild/java_tools/releases/download/java_v11.3/java_tools-v11.3.zip",
  +            "file:///tmp/java_tools.zip",
           ],
           "used_in": [
               "additional_distfiles",

• Build and run the patched Bazel

from java_tools.

I didn't see this issue. I will update docs. Yes, a lot has changed, for example toolchain definitions are not anymore in java_tools.

or is there a better way to do this?

This is the best way to run all the tests.

You could also unzip java_tools and add --override_repository=java_tools=/tmp/java_tools to just use them in a build for arbitrary project.

from java_tools.

Does --override_repository=java_tools=/tmp/java_tools still work? I tried that and it doesn't seem to do anything.

The approach in #43 (comment) still works for me.

from java_tools.

You could also unzip java_tools and add --override_repository=java_tools=/tmp/java_tools to just use them in a build for arbitrary project.

Does --override_repository=java_tools=/tmp/java_tools still work? I tried that and it doesn't seem to do anything.

@comius ping? It would be nice to have a way to test java_tools changes without editing distdir_deps.bzl and rebuilding bazel. Is --override_repository= still a supported way to do that? Maybe I'm just holding it wrong, but it doesn't work for me.

The following does not pick up any local modifications to java_tools:

bazel build //src:java_tools_zip
unzip -q -o -d /tmp bazel-bin/src/java_tools.zip
bazel build --override_repository=java_tools=/tmp/java_tools //src:bazel

from java_tools.

The mechanism should still work, except that you need to override both java_tools and java_tools_linux repositories.
You build zips with bazel build //src:java_tools_zip and bazel build //src:java_tools_prebuilt_zip.

from java_tools.

Facing the same issue. Unfortunately, your workaround @cushon doesn't seem to work anymore (we're on Bazel 5.2). Getting the following error. Does the workaround still work for you?

ERROR: /Users/oliviern/Uber/android/build/.bazelw_cache/91a99194e71852b0e089ca5cc1c48073/external/rules_java/toolchains/BUILD:116:14: no such target '@remote_java_tools_darwin//:ijar_prebuilt_binary': target 'ijar_prebuilt_binary' not declared in package '' defined by /Users/oliviern/Uber/android/build/.bazelw_cache/91a99194e71852b0e089ca5cc1c48073/external/remote_java_tools_darwin/BUILD and referenced by '@rules_java//toolchains:ijar_prebuilt_binary_darwin'

from java_tools.