Regression in `bench_update_many_child_layers.canvaskit.drawFrameDuration.outlierRatio` about flutter HOT 7 OPEN

There was another regression that was reported for that roll here: https://flutter-flutter-perf.skia.org/e/?begin=1719260348&end=1719335402&keys=X9bbdf72b97d78666cbdd3111fa2634b5&selected=commit%3D41333%26name%3D%252Carch%253Dintel%252Cbranch%253Dmaster%252Cconfig%253Ddefault%252Cdevice_type%253DMoto_G__4_%252Cdevice_version%253Dnone%252Chost_type%253Dmac%252Csub_result%253Dcompute_iteration%252Ctest%253Dmicrobenchmarks%252C&xbaroffset=41333

Jason identified https://dart.googlesource.com/sdk/+/452de7b0980a76917a6c0a4669bf3e68f645d00d as the point where the regression started and the Dart team was notified.

from flutter.

@gaaclarke The first regression listed here bench_update_many_child_layers.canvaskit.drawFrameDuration.outlierRatio is in Flutter for Web or Native Flutter ?

from flutter.

@gaaclarke The first regression listed here bench_update_many_child_layers.canvaskit.drawFrameDuration.outlierRatio is in Flutter for Web or Native Flutter ?

That's in flutter web.

from flutter.

Tried bisecting engine builds in that range.

This benchmark is noisy, but the benchmark change appears to be correlated with flutter/engine@e41a8c2 (which contains https://dart.googlesource.com/sdk/+/1c534e5fd5e5c001ac586bbac9e7e3e20569ddcd)

@srujzs

from flutter.

Hmm, I can't really tell how that CL could translate to the resulting benchmark regression, but I'll need to try it locally. I thought maybe the variance is higher due to a lower average, but the benchmark for average hasn't changed (drawFrameDuration at least, totalUiFrame might have decreased however?). Is the next step here for me to revert?

from flutter.

In my local runs I saw significantly higher values for bench_update_many_child_layers.canvaskit.drawFrameDuration.outlierAverage with that Dart roll.

But the average on that same benchmark did not show a major change after the Dart roll.

So the increased outlierAverage versus the similar average means a higher outlierRatio.
(I haven't worked with the internals of these web benchmarks before and have not tried capturing the full set of frame times. Possibly something changed in the distribution of frames that are classified as outliers.)

AFAIK there have not been other alerts on Flutter's performance dashboard associated with this roll. And in my local runs of web_benchmarks_canvaskit I did not notice any other regressions in the metrics.

So a revert may not be necessary - so far I have not seen evidence of a broader issue. But I'd be curious to understand why that change would affect this particular workoad.

from flutter.

So a revert may not be necessary - so far I have not seen evidence of a broader issue.

Got it. Let me know if that changes.

But I'd be curious to understand why that change would affect this particular workoad.

Yeah, I'm not seeing anything that could tie the two together. I'd need to determine where calls from the benchmark flow to .toJS calls. The only thing I can think of that might lead to a greater variance is if for some reason the benchmark ends up calling different arity callbacks across runs. The Dart change speeds up performance for 0-5 argument callbacks much more than it does 5+ argument callbacks. But then we'd see a lower average and not just a higher outlier ratio, so I doubt that's the case.

from flutter.