pyroscope-io / docs Goto Github PK

Describe the process and environment variable used by adhoc in https://pyroscope.io/docs/server-api-reference

Right now all the blogpost show Jan 2nd as creation / publication date in https://pyroscope.io/blog/

The homepage is incredible slow on firefox.

Initially I assumed it was a problem with an extension, but @abeaumont confirmed it also happens on his computer. He also said CPU usage goes to 100%, he thinks must be something with video rendering.

Right now the example flamegraph in our grafana docs is super old and not very interesting. We should update the grafana integration docs to do a few things:

1. Replace the "boring" flamegraph with a much more interesting flamegraph there
2. It would be nice to show the flamegraph in the context of an entire dashboard rather than just a single flamegraph
3. Maybe link to a publish dashboard where users could see the grafana plugin in action
4. Maybe a video of how to install
5. Link to the blog post about grafana plugin

Few things I noticed while setting up in a new machine:

make build-third-party-dependencies

is such a hindrance (you gotta install a bunch of stuff), and for a lot of people contributing to pyroscope it's not really needed. Maybe we should make the ENABLED_SPIES=none more proeminent.

error [email protected]: The engine "node" is incompatible with this module. Expected version "^12.20 || ^14.15 || ^16". Got "17.9.0"

We have to tell people to use nvm (or any other kind of version management like asdf) and follow the node version set up in the .nvmrc file

Running lerna bootstrap is required, otherwise the webapp fails trying to resolve @pyroscope/flamegraph

21:11:58           frontend | @pyroscope/webapp: ERROR in ./javascript/pages/ContinuousSingleView.tsx 5:0-59
21:11:58           frontend | @pyroscope/webapp: Module not found: Error: Can't resolve '@pyroscope/flamegraph' in '/home/eduardo/work/pyroscope/pyroscope/webapp/javascript/pages'

Some of the css was broken

Trying again solved it (???)

It took some time to build the first time
I think it was due to the @pyroscope/flamegraph still being built,
Plus it was difficult to tell when things were ready (ie when should i be able to access localhost:4040 ?)

In this fork we add Pyroscope to the abalone project which is a ruby on rails application. It is pretty straightforward to add we should document it on the ruby page as many who are using ruby are likely using rails as well...

We should also add this to our examples folder in the other repo

In #169 we realized that there is room for alternate build systems. See this comment here for more detail.

For now we show two, but should add documentation for the others as well:

• Maven
• Gradle
• Sbt
• Ivy
• etc(?)

We can do this via code snippet vairables here as well as tab groups

Currently for our docs we reference other repos where the versions change frequently. An example of one are where we do this if for our otel packages where the version in docs can be out of sync with the version from the repo itself

It would be nice if instead of hardcoding this version if we did some sort of javascript request that returns the most recent version so that it's always up-to-date without us remembering to update every time

As this is a common question (eg
https://pyroscope.slack.com/archives/C01FJRYENPQ/p1663579577381859?thread_ts=1663577712.177039&cid=C01FJRYENPQ) we should add it to the faq (https://pyroscope.io/docs/troubleshooting-faq/)

We could base off @petethepig's answer:

this can vary greatly depending on the app, if you’re only using it for local profiling purposes it shouldn’t ever go above 10 GBs. With production usage I’d give an estimate of 250 GB for a 1000 clients and 1 month retention.

https://pyroscope.io/docs/server-configuration is missing several options, both in the table and YAML formats, e.g.:

• auth is missing.
• metrics-export-rules is missing in the table.
• tls-certificate-file is missing.
• tls-key-file is missing.
• admin-socket-path is missing.
• enable-experimental-admin is missing (it should be removed maybe?).
• scrape-configs is missing in the table.
• no-self-profiling is missing.

There are several things that we've noted with the eBPF integration:

• ebpf spy didn't work with version 0.7.2 for some reason. It seem to work with the current version, so I didn't investigate further.
• The documented way to run the ebpf spy on linux sudo -E pyroscope connect -pid -1 doesn't work. Double dashes are needed: sudo -E ./bin/pyroscope connect --pid -1
• At some point, ingestion panicked: grafana/pyroscope#781
• We should also update the documentation in the examples folder

I could barely notice there was a version selected there.

https://pyroscope.io/downloads/

Hi team

Goroutine profile will make STW proportional to the number of goroutines. (before go 1.19)

It would be better to add an IMPORTANT NOTE to notify users to use it carefully on the go client page.

I'll raise a PR later if you agree.

Right now in the eBPF page there is a link to this yaml file that can also be useful if you are using kubernetes and not helm. We should add link to it in kubernetes / helm docs too.

• https://pyroscope.io/docs/ebpf/
• https://pyroscope.io/k8s/pyroscope-agent.yaml

Right now we have connect, exec, push, and pull and it may not be immediately obvious when to use each of these modes for pyroscope.

We should either make one page that explains the differences / nuances / use cases for all these or/and do a more in-depth example on the pages themselves of showing them in aciton:

https://pyroscope.io/docs/agent-configuration-connect/#pyroscope-connect

In #169 we realized that there is room for alternate build systems. See this comment here for more detail.

For now we show two, but should add documentation for the others as well:

• Plain Java
• Spring framework
• Vaadin
• Spark
• Grails
• Play
• etc(?)

We can do this via code snippet vairables here as well as tab groups

We support this, we even have an example but we don't explicitly tell how to use it.

Couple things:

• base-url requires a slash at the beginning eg (/pyroscope)
• If using with auth, redirect-url should be set as well

We recently added support for rust and need to add docs for it here as well

https://pyroscope.io/docs/ebpf/

Pyroscope monitors pods through annotations, and finds the corresponding target to pull profiling data.
We are worried that all pods of this watch will have a large performance impact on our cluster. Regarding this aspect, is there any way to configure the scope of monitoring? thereby reducing the performance impact of apiserver and etcd?

It would be nice to have a blogpost explaining the difference between tracing profiling vs sampling profiling, pros/cons etc.

One our users in slack found out that if you don't have access to k8s nodes to install kernel headers, it's possible to build a docker images with these + pyroscope as a workaround.

It would be nice to document it so that other people also benefit from this info.

Source: https://pyroscope.slack.com/archives/C01FJRYENPQ/p1657750085227309?thread_ts=1657701092.332439&cid=C01FJRYENPQ

In #19 I removed a bunch of data that is useful, in favor of making the pull-mode documentation page easier to parse through quickly, find what you need, and have sufficient information to get started.

An example of this is the Profiles Configuration located below. I think we should leave the pull mode page very simple, but add sub-pages that have the following headings moved to their own respective pages:

• Profiles configuration: This is only necessary if you want to scrape other kinds of profiles like goroutines, but we should really double check that this works the way we expect it too and then devote a full page to this if so
• Service Discovery: This is only necessary if you are using Kuberentes so we should move this page to its own sub-page dedicated to using pull mode inside a Kuberentes context

To be added back when this issue is addressed:

### Profiles configuration

> pprof is agnostic to the profile semantics. . . . The interpretation of the reports generated by pprof depends on the semantics defined by the source of the profile
([__pprof authors__](https://github.com/google/pprof/blob/master/doc/README.md#pprof-profiles)).
Pyroscope can pull `pprof` profiles of any type (such as CPU samples, heap allocations, etc) but requires them to be
explicitly described and enabled. Every scrape job explicitly lists which profiles will be pulled in the
`enabled-profiles` parameter. Profiles listed in `enabled-profiles` refer to configurations present in `profiles`.

Default configuration only includes Go `cpu` and `mem` profiles but eventually we will extend the list.
This means that if you want pulling, for example, Go `goroutine` profiles, you will need to explicitly instruct
Pyroscope how to interpret the data:
```yaml
profiles:
  # Profile name that will be referenced from 'enabled-profiles' option.
  goroutines:
    # URL path. Mandatory field.
    path: /debug/pprof/goroutine
    # URL parameters. The value is a string array.
    params:
      seconds: ['10']
    # A profile can include samples of more than one type. For example,
    # Go 'heap' profile includes: inuse_objects, inuse_space, alloc_objects,
    # and alloc_space.
    # Only explicitly listed sample types are retrieved from the profile.
    # At least one sample type should be configured.
    sample-types:
      # Sample type name as specified in the profile.
      goroutines:
        # Fully qualified application name displayed in UI includes sample type:
        # <application_name>.<sample_type>. Optional.
        display-name: goroutines
        # Sample units. Optional.
        units: objects
        # The field specifies how the profile should be aggregated when queried:
        # if the profile represent the current state (for example, memory in use
        # or goroutines dump), 'average' is the most appropriate aggregation.
        # Supported values: average, sum. Defaults to 'sum'.
        aggregation: average
        # If samples represent accumulated value (for example, alloc_space in Go
        # heap profile shows allocated memory in bytes since the program start),
        # the field should be set to 'true'. Defaults to 'false'.
        cumulative: false
        # If values are sampled, the field should be set to true. For example,
        # Go CPU profile: when CPU profiling is enabled, the Go program stops
        # about 100 times per second and records a sample consisting of the
        # program counters on the currently executing goroutine’s stack.
        # If set to 'true', the profile must contain Period duration value.
        # Defaults to 'false'.
        sampled: false

After configuring goroutines profile, it can be added to the enabled-profiles list.

Default profiles configuration:

profiles:
  cpu:
    path: /debug/pprof/profile
    params:
      seconds: ['10']
    sample-types:
      samples:
        display-name: cpu
        units: samples
        sampled: true
  mem:
    path: /debug/pprof/heap
    sample-types:
      inuse_objects:
        units: objects
        aggregation: avg
      inuse_space:
        units: bytes
        aggregation: avg
      alloc_objects:
        units: objects
        cumulative: true
      alloc_space:
        units: bytes
        cumulative: true

I t

It was implemented here grafana/pyroscope#404

But we don't mention anywhere in the docs

Currently we use an external service (workable) to list our job postings.

• Backend Engineer: https://apply.workable.com/pyroscope/j/28E1725C98/
• Frontend Engineer: https://apply.workable.com/pyroscope/j/CCF934937E/

Would like to move that be in this repo instead so everything is in one place similar to this example

We should make this list:

look more like this:

except since we have react here vs in a readme we should make these components that you can click on

Kind of like these except smaller and mor square:

In the https://pyroscope.io/docs/installing-pyroscope-overview/ page

You can see how the image is on top of the sidebar
https://user-images.githubusercontent.com/6951209/185958561-9b83764a-c555-4f58-a9dd-fe760703c26e.mp4

We're starting to run into more issues where people are profiling very large / complex applications. As we've added new features, it's possible for people to get into issues where a single config option can cause for the Pyroscope server to run into issues:

• too many tags
• Scrape interval too short
• Retention threshold too high/low
• "How much overhead does pyroscope cause"
• 2. if the push destination is unresponsive, will the profiling agent spin, consume resources, or crash?
• etc

We should add more information to various pages to help people troubleshoot the issues that they run into more easily. Not sure if this should be one page or on each individual page, but just adding this here as a placeholder so we can address this more thoroughly as time goes on

It's nice to be able to tell visually whether a link leads to an external website or not.
For illustration, which one of these are external links?

We could add an icon on the right side to indicate it's an external link. For example:
https://blog.codecentric.de

From bunpro

https://electrictoolbox.com/add-offsite-link-icon-after-external-links-css/

Sometimes external links change and we don't know about them (specially the ones linking to our main repo)

We already have something similar for internal pages https://github.com/pyroscope-io/docs/blob/main/.github/workflows/url-linter.rb

The idea is to just parse each link, see if it's external, try to access it and report a failure if it returns anything other than 2xx.

A somewhat common error is the nodejs version:

webpack | error [email protected]: The engine "node" is incompatible with this module. Expected version ">=14.17.0". Got "12.22.5"

Another point of failure is when executing the postinstall script

"postinstall": "./scripts/web-postinstall.sh",

We have to investigate this a little bit more, but it's something to do with the failure to execute the sh binary.

https://pyroscope.io/downloads/

grafana/pyroscope#980

As phlare now supports ingesting data from Pyroscope we should add documentation on how to use Pyroscope's remote-write feature to send data to phlare

We should add a section:

using remote write with phlare

Phlare supports ingesting data from pyroscope .... [short background]

here is how you send data to phlare using Pyroscope's remote-write features:

remote-write:
  enabled: true
  targets:
    phlare:
      address: ...
      headers:
        X-Scope-OrgID: ...

cc @korniltsev