Tool to visualize the Copilot metrics provided via the Copilot Business Metrics API (current in public beta)

Home Page: https://ashy-sky-02a7d0403.5.azurestaticapps.net/

License: MIT License

JavaScript 1.99% HTML 1.27% Vue 76.52% TypeScript 18.83% Dockerfile 0.68% CSS 0.71%

copilot-metrics-viewer's Introduction

NOTE: For information on support and assistance, click here.

GitHub Copilot Metrics Viewer

This application displays a set of charts with various metrics related to GitHub Copilot for your GitHub Organization or Enterprise Account. These visualizations are designed to provide clear representations of the data, making it easy to understand and analyze the impact and adoption of GitHub Copilot. This app utilizes the GitHub Copilot Metrics API.

Video

copilot-metrics-viewer.mov

Charts

Key Metrics

Here are the key metrics visualized in these charts:

Acceptance Rate: This metric represents the ratio of accepted lines to the total lines suggested by GitHub Copilot. This rate is an indicator of the relevance and usefulness of Copilot's suggestions.

Total Suggestions This chart illustrates the total number of code suggestions made by GitHub Copilot. It offers a view of the tool's activity and its engagement with users over time.
Total Acceptances: This visualization focuses on the total number of suggestions accepted by users.

Total Lines Suggested: Showcases the total number of lines of code suggested by GitHub Copilot. This gives an idea of the volume of code generation and assistance provided.
Total Lines Accepted: As the name says, the total lines of code accepted by users (full acceptances) offering insights into how much of the suggested code is actually being utilized incorporated to the codebase.

Total Active Users: Represents the number of active users engaging with GitHub Copilot. This helps in understanding the user base growth and adoption rate.

Languages Breakdown Analysis

Pie charts with the top 5 languages by accepted prompts and acceptance rate are displayed at the top.

The language breakdown analysis tab also displays a table showing the Accepted Prompts, Accepted Lines of Code, and Acceptance Rate (%) for each language over the past 28 days. The entries are sorted by the number of accepted lines of code descending.

Copilot Chat Metrics

Cumulative Number of Turns: This metric represents the total number of turns (interactions) with the Copilot over the past 28 days. A 'turn' includes both user inputs and Copilot's responses.
Cumulative Number of Acceptances: This metric shows the total number of lines of code suggested by Copilot that have been accepted by users over the past 28 days.
Total Turns | Total Acceptances Count: This is a chart that displays the total number of turns and acceptances
Total Active Copilot Chat Users: a bar chart that illustrates the total number of users who have actively interacted with Copilot over the past 28 days.

Seat Analysis

Total Assigned: This metric represents the total number of Copilot seats assigned within current organization.
Assigned But Never Used: This metric shows seats that were assigned but never within the current organization. The assigned timestamp is also displayed in the below chart.
No Activity in the Last 7 days: never used seats or seats used, but with no activity in the past 7 days.
No Activity in the last 7 days (including never used seats): a table to display seats that have had no activity in the past 7 days, ordered by the date of last activity. Seats that were used earlier are displayed at the top.

Setup instructions

In the .env file, you can configure several environment variables that control the behavior of the application.

VUE_APP_SCOPE

The VUE_APP_SCOPE environment variable in the .env file determines the scope of the API calls made by the application. It can be set to either 'enterprise' or 'organization'.

If set to 'enterprise', the application will target API calls to the GitHub Enterprise account defined in the VUE_APP_GITHUB_ENT variable.
If set to 'organization', the application will target API calls to the GitHub Organization account defined in the VUE_APP_GITHUB_ORG variable.

For example, if you want to target the API calls to an organization, you would set VUE_APP_SCOPE=organization in the .env file.

VUE_APP_SCOPE=organization

VUE_APP_GITHUB_ORG= <YOUR-ORGANIZATION>

VUE_APP_GITHUB_ENT=

VUE_APP_MOCKED_DATA

To access Copilot metrics from the last 28 days via the API and display actual data, set the following boolean environment variable to false:

  VUE_APP_MOCKED_DATA=false

VUE_APP_GITHUB_TOKEN

Specifies the GitHub Personal Access Token utilized for API requests. Generate this token with the following scopes: copilot, manage_billing:copilot, manage_billing:enterprise, read:enterprise, admin:org.

  VUE_APP_GITHUB_TOKEN=

Install dependencies

npm install

Compiles and runs the application

npm run serve

Docker build

docker build -t copilot-metrics-viewer .

Docker run

docker run -p 8080:80 copilot-metrics-viewer

The application will be accessible at http://localhost:8080

License

This project is licensed under the terms of the MIT open source license. Please refer to MIT for the full terms.

Maintainers

@martedesco

Support

This project is independently developed and maintained, and is not an official GitHub product. It thrives through the dedicated efforts of myself (@martedesco) and our wonderful contributors. A heartfelt thanks to all our contributors! ✨

I aim to provide support through GitHub Issues. While I strive to stay responsive, I can't guarantee immediate responses. For critical issues, please include "CRITICAL" in the title for quicker attention. 🙏🏼

Coming next 🔮

Team slicing
Persistence layer

copilot-metrics-viewer's People

Contributors

Stargazers

Watchers

Forkers

bens-scratch-org andrewgrizhenkov hawaku 7thmarch kentakozuka stephenjannin adriandsg like10 im-infomagnus nagarajumatta tompkyle 0gis0 aseidasvera gergelyke-dd whsalazar-org plentytale-riseraci louud70 yesmessages-trust4me pintaki-flasher saviorrazu0glutilli girlieinkshamp maintemagreator centhart-onettens readyna-o nixonet-6 paybaxanekowolfie generalweare-b giotab rsam444 jivane im-infomagnus moneyforward-i insight-services-apac sjwaight theodoremeynard pankajagrawal16 github-4women msopariw avanade-region-iceg eldrick19 raffertyuy emanuele-ripamonti-ava pietroboccia shoaibm-rakbank solidifydemo cdwbrad schoaib abirismyname krvns a2atsu sifiadlene berkovic95 etn-corp nataliagranato aline-viana wernerdevops ind-ict petermyqops realbroz parul-mahajan ppottorff devops-zhuang bbh88812345 camilacespedes kokodoki alexis-landrieu-avanade nickhou1983 gcwngo nathenharvey uainc anurag-blusmart guildeducationinc naveen1890 devtoolstestmigration cbsi-cloudarch amarnathsr divzi-p msft-appinn-grcymt im-sandbox-tamb andrekolodochka iqmetrix noest-nv abagga1c vishal-shl danielfree roverdotcom cherryjain-msft zdc-naka

copilot-metrics-viewer's Issues

Update access rights for github token?

In the readme the following information is provided.

VUE_APP_GITHUB_TOKEN
Specifies the GitHub Personal Access Token utilized for API requests. Generate this token with the following scopes: copilot, manage_billing:copilot, manage_billing:enterprise, read:enterprise, admin:org.

I believe that admin:org is overkill. When I did my local setup, I used the less intrusive scope read:org, and it worked just fine. If there is no functionality I'm missing, I would suggest that the README be updated to adhere to the principle of least privilege.

Configurable Variables

Hi,

We have encountered a problem where it is currently not possible to pass environment variables to a Docker container from outside. This seems to be due to the way the variables are integrated.

It would be much nicer if we could pass the variables through a separate environment file, such as a config.json.

Then, we could refer to the config.json when running npm serve, and in the Docker context, an environment variable or an env-file could be passed.

This would mean that no variables are required at build time.

Also, this approach would eliminate the need to constantly rebuild the container or run npm install, and it would be possible to provide one's own access token or other Vue variables at container start, without them being stored in the container during the build.

However, this would require adjusting how the variables are processed.

Thanks

Is Seat count available in Enterprise metrics

Is Seat count available in Enterprise metrics?

support for persistent data

As documented, the API contains metrics for the prior 28 days. It would be useful if there would be a way to configure this to support a 'local' json file containing longer-term data (e.g. we currently periodically merge the results into a larger json file with more historical data). Or, perhaps, support an external datasource where the API results are collected and persisted for a longer historical view.

Percentage calculations seem to be off

Can someone please explain how the average calculations are computed?

Looking at the values in the readme for Cumulative Number of Accepted Prompts (50734) and Cumulative Number of Suggested Prompts (199097) I would expect the average would be 50734 / 199097 * 100 or 25.48% whereas the image shows 22.19%

I ran this locally again my Enterprise and the values are 20024 and 68238 so I would expect an average of 29.34% yet the value given is 20.54%

I would like to understand where these differences are coming from so that I can make sure I have the correct values.

Logo link is broken

the Logo image link is broken because there is no assets folder

Support for date range pickers

Following the addition of persistent storage, it will likely be helpful for users to have the ability to select start and end dates for the data visualizations on each page.

Filtering on projects, developers, teams

It would be useful to filter metric results on users, projects, teams. I don't see a current way of doing this. Thoughts?
Thanks #

Support for connecting via GitHub App

In the .env file, there is an option to connect with a GitHub Personal Access Token:

VUE_APP_GITHUB_TOKEN=

In large organizations, there are policies against over granting privileges to tokens for sensitive things like copilot billing and usage. Following the principle of least privilege, we are required to access sensitive data via a GitHub app.

Support for connecting through a GitHub app would allow us to use this project as is. For example:

VUE_APP_GITHUB_KEY=<name of the private key file>
VUE_APP_GITHUB_APPLICATION_ID=<application ID>
VUE_APP_GITHUB_INSTALLATION_ID=<installation ID>

A workaround is to extract the data from the GitHub Copilot Usage endpoint, and store it in a file called
src/resources/metricsExample.json and set VUE_APP_MOCKED_DATA=true. This would require additional automation, and the preferred way would be to pull real time data from the API with a GitHub App.

{
        "day": "2024-04-23",
        "breakdown": [
            {
                "language": "java",
                "editor": "jetbrains",
                "suggestions_count": 56,
                "acceptances_count": 16,
                "lines_suggested": 99,
                "lines_accepted": 20,
                "active_users": 2
            },
            {
                "language": "kotlin",
                "editor": "jetbrains",
                "suggestions_count": 27,
                "acceptances_count": 8,
                "lines_suggested": 41,
                "lines_accepted": 7,
                "active_users": 2
            },

can 28 days be changed? or expanded?

README Seat Analysis image update

          We might need to update this image once the changes have been implemented 🙏🏼

Originally posted by @martedesco in #32 (comment)

consider building and hosting docker image

Hi,
I just discovered this great project.

Please consider building and hosting the docker image using GitHub Actions and the GitHub Container Registery (GHCR).

It would ease the usage of the project significantly.

github-copilot-resources / copilot-metrics-viewer Goto Github PK