Separate tests from builds that generate artifacts about flutter HOT 10 OPEN

Seems reasonable, but what needs to be done here? I don't think we combined the builders intentionally, so much as we started with limited capacity (and tests) and so everything ran on one machine.

Is there a way to build everything on one shard an execute the tests on another (so we can avoid building things twice)?

from flutter.

@christopherfujino added in chat:

My understanding is what is needed here is @keyonghan's graph scheduler, is that right?

I added:

Maybe, but I am hoping we don't build a dependency graph around how to do the underlying work

That is, I am hoping the answer is "Separating the tests manually is not needed, carry on" or "Graph scheduler will take N quarters, so let's separate them manually for now"

from flutter.

The goal here is to separate tests from building artifacts as different builder targets.

Take Mac mac_host_engine for example

• Now: the test Host Tests for host_debug is running within the same subbuild which builds the artifact.
• After: a new .ci.yaml builder target to be added to focus on running the tests

Some TODO work:

- [ ] Infra team
- [ ] recipes support to have a way to reference artifacts generated in the build targets
- [ ] add docs explaining how to add these targets in .ci.yaml

• Engine team
  • separating the tests as a separate target in .ci.yaml

Long term: the graph scheduler should be able to handle the build and test scheduling efficiently enough that no new targets are needed. Say: the scheduler schedules build as target A, and schedules test as target B, without concatenating them inside a single target. This way, we can control rerun of the test separately. (@matanlurey there does exist some overlap considering long term:-))

/cc @godofredoc please correct me if I miss anything.

from flutter.

@keyonghan My understanding is the infra work is blocking the second part, the engine team work?

If so, let's re-assign this to team-infra. Thanks!

from flutter.

IIUC, we can do this today without any recipe changes with the downside that some builds will be duplicated. In particular, builds that both produce artifacts and run tests would be split into two builds. One build that only produces artifacts, and a second build that is only done for the tests. So for example, we'd split mac_host_engine.json into mac_host_engine_artifacts.json and mac_host_engine_tests.json. mac_host_engine_artifacts.json would not run any tests, and mac_host_engine_tests.json would not upload any artifacts.

As for some of these builds being duplicated: (1) The builds that produce artifacts are already not using goma or RBE on the release builders, but the since the builds for testing aren't producing artifacts, they could use goma/RBE, so the additional builds will be much cheaper. (2) The ideas mentioned above for recipe changes could eliminate the duplicate work as an optimization.

from flutter.

If we the capacity to do that, then we can definitely make the changes

from flutter.

but I guess I don't understand how that will help with the problems of reruns being slow on release branches: they'd still end up building all of the artifacts?

from flutter.

The builds whose artifacts aren't retained and used for the release don't need to be done on dash-internal, and can use Goma/RBE.

from flutter.

Ahh okay, so the problem is not just rerunning the builders, its that the secure builders are slow

from flutter.

Yeah, that's doable, but will build the artifacts twice in separated .json builders.

For release branch: the execution time of the test one will be reduced considering using rbe, but will consume extra (non dart-internal) bots
For main branch: same as release branch, this will consume more capacity for each run

The queue time in our pool has been well within SLO for a while. But this may potentially increase the queue. It would be good we incrementally refactor targets.

(Considering this is not blocked on recipes, routing back to engine team for config refactoring)

from flutter.