This epic is a catch all for the MVP flamegraph features and design doc containing the investigative takes taken prior to deciding what features we want for the MVP.

• Design doc
• Base flamegraph visualisation #146
• Flamegraph navigation #149
• Flamegraph search #159
• Flamegraph tooltips

When zooming in chrome the view breaks and the right is hidden

I can't click on run queries and I don't fully see the flamegraph

In explore for tempo this is fine though

currently this is only set in the distributor and not useful for deduping back.

The whole flamegraph is currently not embedded into box panel like logs for instance. see below

The box panel should contains the following:

• Searching symbols.
• Reset node selection.
• #173
• grafana/grafana#58378
• #171
• grafana/grafana#58377

Some of those might be better to be part of the flamegraph plugin directly TBD.

• counter of total received bytes compressed/uncompressed.
  • counter of profiles types per type and name.
• histogram for size compressed/uncompressed

We should add a way to query for top table I suggest for now we do this in computation in the datasource and add it to the frames.

This should produce an array of self/total per symbols including root.

Part of: #147
Use APIs implemented in #142 to being able to query data.

The basic UI should show:

• timeseries of the sum of values by labels set.
• flamegraph: the merge stackaces as a by default over the timerange

Each point within the timeseries should allow to see a sigle point in time profile as a flamegraph.

Top tables are an important part of profile visualisation, We should definitively add a way to open a persistent drawer to see the top symbols by ordered by top %self or %total.

Related backend work #172

We should to always keep full visibility if possible of the flamegraph by resizing it this way we can select top table symbols and highlight the flamegraph. In the case resizing the flamegraph is not possible if the screen is too small then replacing the visualisation will be done.

Example from Pyroscope

Stacktraces labels account for a big part of the in-memory usage whereas currently they are actually every-time the same.

see

We should find a way to reuse the pointer instead of creating a new one every time.

When looking at the framegraph and refreshing the flamegraph ordering may changes this is probably introduce in the querier code when building the flamebearer.

see https://github.com/grafana/fire/blob/main/pkg/querier/querier_test.go#L153

This is a follow up on #19 and to investigate migrating the target API to the connect framework.

Like mimir https://github.com/grafana/mimir/blob/main/pkg/api/api.go#L413

• Have a grafana instance with the explore hack

Using OTel as much as possible.

The current colouring is focusing on the total of nodes but not the self making it actually upside down as opposed to what is expected.

Expected from pprof

In a node there's always 2 values, the total and the self, the total includes the self + all children (self).

What's interesting in pprof is that they actually do that per function, in the example you can see some self of main.fibhandler are small but still in red.

I think it's important to get the coloring right since this is what catches the attention to the user.

https://www.brendangregg.com/blog/2017-07-30/coloring-flamegraphs-code-type.html is a really good read before tackling this issue.

We need a tool to automatically check and apply OSS license for the project, it should be accessible via the Makefile.

• Check for go.mod similarly like for fire itself #163
• Run tests and build

This would allow to slice the data further, the data should be ordered first by type and by timestamp.

On each commit we should deploy the new version of the plugin to a grafana instance in dev (could be a new one or https://admin-dev-us-central-0.grafana.net/grafana/?orgId=1)

The plugin should be updated and reinstalled.

This probably means we need to a run a fork of grafana for now see grafana/grafana#52057

Allow the user to add label names to group by in the series graph

• Will use the new labels API
• Let user select label name via dropdown

When building a dashboard if you use the Fire datasource and query, by default it uses the TimeSeries Panel see below

But by default we should automatically select the flamegraph viz

We need a way to show the user the variation over time for the current query, this allows them to then drill down to the right range they are interested into.

Backend API PR: #160

This should be on top of the flamegraph and if possible reusing the grafana series panel.

Unlike pyroscope I'd like us to try to avoid bar but instead default to lines like for timeseries in the graph panel and then allow to swap with bar,point,....

The panel should provide:

• the ability to have a sense of the value and variation using y axis labeling and may be even a title.
• the ability to zoom in and out via selection and change the timerange of explore.
• Select via dropdown labels name to group by (see the API)
• Handle gracefully if there is too many series returned.
• Select and highlight a timeseries
• Add a new label selector based on a selected series in the query

• Use approximation rather than sizeOf calls
• Expose via gauge metric (potentially using atomic.uint64

Have a grafana instance with the explore hack

It seems cheap to save into the Profile schema the total value for each profiles.

This would make it simple for building histogram. Specially with parquet we could load that column only when needed.

....

When reading pprof a lot of stacktrace samples contains values of zero and are forcing us to use more resources on the read/write path.

I suggest we remove those in the distributors before pushing to ingesters.

When hovering over a flamegraph node we should present to the user a popup that contains more details about the current node such as:

• the full symbol name
• the total and self amount in the right sample type.
• The percentage of both
• A way to copy the symbol name and highlight other node with the same name.

Example from pyroscope:

The datasource could detect if this is a Fire or Parca datasource in which case it could use a different API to fetch data.

This would allows to send profile in order and so ease the streaming.

Initial implementation of the fire datasource.

• Implement basic querying support #144 #148
• Implement autocomplete for the editor for labels/values #155
• Changing the dataFrame format to remove need for json strings in dataframe #215
• Support query for visual histogram of profile samples
• Table view data
• Streaming query support
• Visual query builder [TBD]
• Profile diff query [TBD]

The pprof memory profiles is different compare to other profiles as it contains all allocations since the beginning, plus the current in use heap. This means the profiles gets bigger over time and we're currently storing duplicate data.

Instead of storing the whole profile every-time we actually should diff allocations (bytes+count) and store only that. My first reaction was that we should do that in the client and only send the diff. But if the client is restarting or rolling often then it won't be able to send the diff.

So I'm suggesting that:

• First, we implement a diff on the ingester side using the last seen profile for that label set and only store the diff. (Diff memory profiles #124
  )
• Clean up periodically samples that are not seen anymore. see comment
• Then, we also implement a diff on the client side, the client side diff will be contains a specific label when computed (e.g __name__="memory_diff"), when the ingester spot that label he won't compute the diff and store directly the diff.

I think we can leave the client side for later as we're unsure what we will do in the future.

• Deploy tracing at full sampling rate

the helm chart should only deploy fire as a single binary (target=all) with PVC

We could actually just implement the Push service and register it on the ingester side.

This will avoid multiple push definitions and make the proto linter happy

I think we should use a monospace font for the flamegraph. Right now it's Roboto. We could swap it to Roboto Mono ?

Roboto:

Roboto Mono

May be that means decreasing the font-size too ?

With regards to the text I think we should also include % and total of the node in its respective sample type like pyroscope

This would allow to have contiguous data per profile.

• Implement flush endpoint for manual writing

• The docker image should default to single binary.
• Add the necessary Makefile command build and push the image.
• Hook with drone.grafana.net