Presubmit builds may check out the wrong commit about flutter HOT 21 CLOSED

FYI @QuncCccccc and @TahaTesser

from flutter.

I filed a duplicate for #149739 and it has some specific info about the failure there that might be helpful: #149928

from flutter.

Working on getting the failing post submit builds for the others, they were not included in the revert

from flutter.

Just to add another case, this issue happened again in this PR: #149739

from flutter.

Also, it appears that it checked out the wrong commit only on the rerun. The original presubmit run for that builder actually did check out the correct commit and failed (as expected): https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20web_tests_0/67329/infra

from flutter.

For the PR: #149245

We scheduled the try build (which started at June 6 01:15AM PDT): https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20web_tests_2/67064/infra

Which, in step 3.4 appeared to check out commit 4608a8913767116234e95cb7d73a410e9a7a2fc9, which was merged to master on June 5, 11:11PM PDT.

from flutter.

For the PR: #149245

We scheduled the try build (which started at June 6 01:15AM PDT): https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20web_tests_2/67064/infra

Which, in step 3.4 appeared to check out commit 4608a8913767116234e95cb7d73a410e9a7a2fc9, which was merged to master on June 5, 11:11PM PDT.

I also see in the input properties: "git_branch": "master": https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20web_tests_2/67064/infra. I'm thinking maybe the bug is in the input properties Cocoon is scheduling these builds with?

from flutter.

Actually, comparing it with this failed try build: https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20web_tests_2/67193/infra, it has a "git_ref": "refs/pull/143249/head" field, so I think the fact that our mistaken build is missing that field is why we checked out the wrong commit.

from flutter.

It looks like Cocoon populates this when generating its BuildBucket RPC request here

from flutter.

Also, it appears that it checked out the wrong commit only on the rerun. The original presubmit run for that builder actually did check out the correct commit and failed (as expected): https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20web_tests_0/67329/infra

Ahh, I wonder if re-runs go down a different codepath...

from flutter.

Although, it looks like when Cocoon receives the request from a GitHub check-run to retry a build, we call the same function (which should populate properties.git_ref): https://github.com/flutter/cocoon/blob/main/app_dart/lib/src/service/luci_build_service.dart#L485

from flutter.

I see this in the cocoon logs:

> INFO 2024-06-06T08:14:56.219Z

Making bbv2 request with path: https://cr-buildbucket.appspot.com/prpc/buildbucket.v2.Builds/ScheduleBuild and body:

{
  "builder": {
    "project": "flutter",
    "bucket": "try",
    "builder": "Linux web_tests_2"
  },
  "properties": {},
  "tags": [
    {
      "key": "buildset",
      "value": "pr/git/149245"
    },
    {
      "key": "buildset",
      "value": "sha/git/49f9a3b3ce6fa06b7c7fd6f91de7533d132cacf1"
    },
    {
      "key": "cipd_version",
      "value": "refs/heads/main"
    },
    {
      "key": "github_checkrun",
      "value": "25879982459"
    },
    {
      "key": "github_link",
      "value": "https://github.com/flutter/flutter/pull/149245"
    },
    {
      "key": "user_agent",
      "value": "flutter-cocoon"
    },
    {
      "key": "current_attempt",
      "value": "2"
    }
  ],
  "notify": {
    "pubsubTopic": "projects/flutter-dashboard/topics/build-bucket-presubmit",
    "userData": "ZXlKaWRXbHNaR1Z5WDI1aGJXVWlPaUpNYVc1MWVDQjNaV0pmZEdWemRITmZNaUlzSW1Ob1pXTnJYM0oxYmw5cFpDSTZNalU0TnprNU9ESTBOVGtzSW1OdmJXMXBkRjl6YUdFaU9pSTBPV1k1WVROaU0yTmxObVpoTURaaU4yTTNabVEyWmpreFpHVTNOVE16WkRFek1tTmhZMll4SWl3aVkyOXRiV2wwWDJKeVlXNWphQ0k2SW0xaGMzUmxjaUlzSW5KbGNHOWZiM2R1WlhJaU9pSm1iSFYwZEdWeUlpd2ljbVZ3YjE5dVlXMWxJam9pWm14MWRIUmxjaUlzSW5WelpYSmZZV2RsYm5RaU9pSm1iSFYwZEdWeUxXTnZZMjl2YmlKOQ=="
  }
}

> INFO 2024-06-06T08:14:56.725Z bbv2 request returned response code 200

Which comes from here in the Cocoon source.

And the response was:

bbv2 request response body: )]}'
{
  "id":"8745894885153659393",
  "builder": {
    "project":"flutter",
    "bucket":"try",
    "builder":"Linux web_tests_2"
  },
  "number":67064,
  "createdBy":"user:[email protected]",
  "createTime":"2024-06-06T08:14:56.248205349Z",
  "updateTime":"2024-06-06T08:14:56.248205349Z",
  "status":"SCHEDULED",
  "input":{}
}

from flutter.

Meanwhile, from the LUCI milo log of the actual scheduled build, these were the input properties:

{
  "$flutter/goma": {
    "server": "rbe-prod1.endpoints.fuchsia-infra-goma-prod.cloud.goog"
  },
  "$flutter/rbe": {
    "instance": "projects/flutter-rbe-prod/instances/default",
    "platform": "container-image=docker://gcr.io/cloud-marketplace/google/debian11@sha256:69e2789c9f3d28c6a0f13b25062c240ee7772be1f5e6d41bb4680b63eae6b304"
  },
  "$kitchen": {
    "emulate_gce": true
  },
  "$recipe_engine/isolated": {
    "server": "https://isolateserver.appspot.com"
  },
  "$recipe_engine/swarming": {
    "server": "https://chromium-swarm.appspot.com"
  },
  "bringup": false,
  "clobber": false,
  "cores": 8,
  "dependencies": [
    {
      "dependency": "android_sdk",
      "version": "version:34v3"
    },
    {
      "dependency": "chrome_and_driver",
      "version": "version:119.0.6045.9"
    },
    {
      "dependency": "goldctl",
      "version": "git_revision:720a542f6fe4f92922c3b8f0fdcc4d2ac6bb83cd"
    },
    {
      "dependency": "curl",
      "version": "version:7.64.0"
    }
  ],
  "device_type": "none",
  "git_branch": "master",
  "git_repo": "flutter",
  "gold_tryjob": true,
  "goma_jobs": "200",
  "mastername": "client.flutter",
  "os": "Ubuntu",
  "presubmit_max_attempts": 2,
  "rbe_jobs": "200",
  "recipe": "flutter/flutter_drone",
  "shard": "web_tests",
  "subshard": 2,
  "tags": [
    "framework",
    "hostonly",
    "shard",
    "linux"
  ],
  "upload_packages": false,
  "use_cas": true
}

from flutter.

After you get the root cause fixed consider failing a run if "git_branch" is not set.

from flutter.

After you get the root cause fixed consider failing a run if "git_branch" is not set.

I will consider it, although the constraints are quite complex

from flutter.

Jason merged flutter/cocoon#3777 which we believe will fix this. However, it has not been deployed yet. Looking into making a manual deploy.

from flutter.

Would this bug cause reverts merged into master to not appear on the build dashboard?

from flutter.

Would this bug cause reverts merged into master to not appear on the build dashboard?

No, I don't believe so. I'm guessing you're observing that @yaakovschectman?

from flutter.

@christopherfujino Yes. You can see on the commit history - https://github.com/flutter/flutter/commits/master/ - that a revert landed 4 hours before this comment, but the build dashboard does not go beyond aad70e5, which the most recent (as of now) commit reverted

from flutter.

@christopherfujino Yes. You can see on the commit history - https://github.com/flutter/flutter/commits/master/ - that a revert landed 4 hours before this comment, but the build dashboard does not go beyond aad70e5, which the most recent (as of now) commit reverted

Can you file a new issue for that?

from flutter.

Per my testing on #150061, I believe this is fixed. Please ping me ASAP if you observe a presubmit build running a commit from the master branch.

from flutter.