squarewave / bhr.html Goto Github PK

We would need to upload the "profile" somewhere (potentially just overloading perf.html's uploads), and create a URL for the current UI state.

• mstange

It would possibly be helpful to filter stacks to exclude names, such as those containing "sleep" or "wait" that are likely uninteresting in themselves.

Historical view was running into memory issues, so I had to break it up per thread. The UI just needs to be adjusted to fetch these new threads when you change between them, rather than having them all there and ready.

Digging into the frames we very quickly get down to 0%, it might be helpful so show these in scientific/exponential notation as 1e-4, etc.

Right now in the profile format we have a table of hang milliseconds and hang counts for each date for which we collected information. Each of these tables has the same length as the stackTable, with each row corresponding to a row in that table. Since the data is actually fairly sparse, with hang milliseconds only properly belonging to the leaves of the stack tree, we probably take a pretty large performance hit from storing it this way. We should consider a different format. Perhaps an intermediate "samples" table, which has only those stacks which have hang information directly associated with them, and then the dates tables can point into this?

Right now because of OOM issues in the job and performance issues in the UI, we're on a 2% sample size of hangs. This makes things much noisier and less stable than they should be, so we should try to bump it up to at least 10%, but if possible it would be ideal to go all the way to 100%.

I would like to start tracking some specific stacks related to DevTools.
But I'm not sure they will be all worth tracking and I may have many to track.
I was wondering what was the machinery behind the "tracked hangs" section?
Is this something I can setup locally on my machine?

Right now there's a blank(ish) page that just says "Waiting for data from telemetry..." and never goes away. In the actual payload that we've processed we see just "{"threads": []}". Further investigation reveals that this was just a bug in get_pings_properties, for which there is already a PR.

It would be super awesome if this could be like the categories dropdown, and just filter everything by which platform the hang occurred on (e.g. win32, win64, linux64, etc...)

I imagine it might be more useful to group the hangs by day of nightly build, as that is more likely to be related to the actual hangs than the day of submission would be.

I'm not sure what the current implementation does, but I would guess it's day of submission (not sure).

I was just looking at https://bugzilla.mozilla.org/show_bug.cgi?id=1399851 where we have a wide variety of crashes (crash-stat link) that were triggered at shutdown because shutdown took too long and it looks like it's a massively manual process to figure out if there are common problems/low hanging fruit/etc.

It strikes me the bhr.html UI is perfect for this problem and crash-stats can already provide the thread backtraces as JSON. It also strikes me that maybe we already have BHR reports for these hangs and so maybe we don't need to look at crash-stats?

I'm filing this issue because I'm not sure where to best raise these questions:

1. Does BHR already have reports for these hangs and the crash reports should be ignored?
2. If not, would it be cool / a sane idea to try and make a hacky PoC to use this UI and backend to digest that data?
3. Has some awesome person already started doing this? Or is there an existing awesome person who already views this as part of their day job and can get to it soon? ;)

Thanks much and thanks for the awesome tool! (Aside: is there a better domain name for me to use than squarewave.github.io? Like a glamour arewenothangingyet.com?)

There are occasionally spikes in the graphs that seem to show up for no particularly good reason. We should analyze why these show up and then take appropriate action to filter them out if they don't provide meaningful information.

We should include documentation of our hangs pseudo-profile similar to the documentation that exists in perf.html.

I've notice when looking at the code a bit that especially in the content process, (content script) entries make up a very large percentage of the long hangs. These are frustrating as they are not really actionable (it's slow because code which we can't control is being slow), so I would like to be able to hide them when looking at BHR data.

When looking at the hang information it's fairly easy to drill down and look at a single category of hangs, but it isn't as easy to filter categories of hangs out of the view.

For example, the majority of hang stacks in most views are exclusively gc or cc hang stack frames. When I'm looking for other issues than "gc/cc is slow", it would be useful if I could hide all of the gc/cc hang information entirely.

Is that expected?
Do we have a delay in data processing?

For example, when I open past 9 days for light hangs:
https://arewesmoothyet.com/?mode=explore&durationSpec=128_512&historical=false
I get results from Nov 19th to Nov 26th, whereas we are today Dec 5th.
I would expect to see from Nov 27th to today, Dec 5th.

We're currently using Chart.js, which has a nice set of features and was quick to use, but it's not really built for our use case and it would be nice to have more flexibility. Here's a list of things that we should do which would be hacky / not possible with chart.js:

• Show a legend item for each category and for each hang duration, (i.e., the number of legend items would be (num_categories + num_hang_durations) rather than (num_categories * num_hang_durations)
• Persist the checked state of each legend item, and encode it in the URL, so users don't have to recheck everything they're interested in every time they visit the page.
• Keep the height of the graph consistent when there are large numbers of legend items (i.e., don't shrink the graph when there are 20 legend items)
• Add marker lines so we can show bug numbers for known regressions/fixes that show in the graph
• Make a better mouseover-tooltip that shows each category for a given x-position

See for example: https://arewesmoothyet.com/?mode=track&trackedStat=Devtools%20Hangs

Loading failed for the <script> with source “https://arewesmoothyet.com/analytics.js”.
arewesmoothyet.com:10
Unhandled promise rejection 
TypeError: e.parentNode is null
Stack trace:
a@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:11:20222
dangerouslyReplaceNodeWithMarkup@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:41:57647
_replaceNodeWithMarkup@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:41:69487
_updateRenderedComponent@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:41:69411
_performComponentUpdate@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:41:68834
updateComponent@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:41:68112
receiveComponent@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:41:67229
receiveComponent@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:11:23362
_updateRenderedComponent@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:41:69080
_performComponentUpdate@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:41:68834
updateComponent@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:41:68112
performUpdateIfNecessary@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:41:67445
performUpdateIfNecessary@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:11:23544
s@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:11:2952
perform@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:11:47413
perform@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:11:47413
perform@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:11:4004
T@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:11:4181
closeAll@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:11:47995
perform@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:11:47493
batchedUpdates@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:41:94544
u@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:11:3165
r@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:11:78802
enqueueSetState@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:11:79931
r.prototype.setState@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:36:1800
l/</r</s.prototype.handleChange@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:41:139877
d@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:36:3937
r/</</<@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:36:7065
r/</</<@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:36:7065
r/</</<@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:41:164041
dispatch@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:41:164346
t/<@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:36:13608
r@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:41:167497
l/<@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:41:168619
s/</e[t]@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:41:167673
r@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:36:125510
r/<@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:36:125610
C/</<@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:36:277595
C/<@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:36:277465
l@https://arewesmoothyet.com/cc8c2b64f45ad5919d95.bundle.js:36:272183
cc8c2b64f45ad5919d95.bundle.js:36:277934
[SW]: Install event
sw.js:3:6891
Error: Wrong response status
sw.js:3:2452
Service worker event waitUntil() was passed a promise that rejected with 'Error: Wrong response status'.
sw.js:3:5199

We should be able to file a bug by right clicking on a particular frame. It should look at the stack and fuzzy search Bugzilla for any existing bugs that might be associated with that stack.

See the ohnoreflow's solution which should be similar to what we need.

It seems like all of the hangs on the compositor thread are labeled as (other), making the stacks almost completely useless.

There's already code for this in perf.html that was stripped out as part of the initial port. Should be able to just wire it back up.

• Merge node into calling function
• Merge function into caller across entire tree
• Focus on function
• Collapse all in library

You can click on the categories tab and select uncategorized, but it doesn't appear in the category: dropdown.

It would be nice if we could use the tools which we have in the stack view to filter which stacks are displayed to also filter the views of stacks in the runnable/category views. For example, it would be nice to have a runnable overview page which shows the top runnables on a specific day when the user is interacting.

Severity levels should be as follows:

low: hangs > 128ms
medium: hangs > 512ms
high: hangs > 2.5s (currently due to the histogram-based nature of BHR data, we can only do hangs > 2048 ms, but the format should be changing relatively soon)

For some reason, when I select the ??? runnable name from the Runnables tab, it shows no hang stacks associated with that runnable in the Call Tree.

It would be nice if we could make looking up runnables with the unknown name work correctly.

See https://bugzilla.mozilla.org/show_bug.cgi?id=1377344 for more information.

I'm not sure how possible this is, because I suspect that any add-on stacks we get will have profile-specific moz-extension URIs, which... at least from what I can tell from the Telemetry payloads, isn't super easy to map back to add-on IDs.

But I think that'd be useful to have, and to share with add-on authors.

This will allow users to see historical trends for a particular criterion without having to wait for a long time for the normal, fully explorable historical data (which also might crash the content process.) The data is currently at https://analysis-output.telemetry.mozilla.org/bhr/data/hang_aggregates/historical_data.json, so we just need to build a viewer for it.

It would be helpful to understand in the BHR data what hangs are occurring only while the user is interacting. It might be best to actually make this the default, rather than a special filter.

If you look at this graph:
https://arewesmoothyet.com/?mode=track&trackedStat=Devtools%20Hangs

The latest update is on June.
Could it be that the cron job is broken, or are you triggering this job manually?