github-copilot-resources / copilot-metrics-viewer Goto Github PK

Hi,

After deployment on Azure the website is not working, URL: https://ashy-sky-02a7d0403.5.azurestaticapps.net/

Get the following error in console:
Error: Invalid VUE_APP_SCOPE value: ${VUE_APP_SCOPE}. Valid values: organization, enterprise

This used to work around a month ago, but when I pulled latest changes a few days ago it stopped working.

Is Seat count available in Enterprise metrics?

Is the image generated by #27 available publicly? I seem to get an unauthorized:

sudo docker pull ghcr.io/github-copilot-resources/copilot-metrics-viewer:latest
Error response from daemon: Head "https://ghcr.io/v2/github-copilot-resources/copilot-metrics-viewer/manifests/latest": unauthorized

Hi @martedesco,

GitHub has finally share that they have released the API for team-level GitHub Copilot metrics (beta).

You can find all the information in our changelog post here - https://github.blog/changelog/2024-08-09-copilot-metrics-api-organization-team-metrics

Any plan to update it in "copilot-metrics-viewer" and release the new version 1.7

As we are adding more features for the github copilot usage API, we found the data returned by the copilot API are not always complete. so I summary the below data missing scenarios, and are considerring exposing such missing in the copilot view tool. is it needed?

Data missing scenarios:

1. No data for some day, even it is within past 28 days.
2. total_lines_suggested are zero, even there are datas.
3. the breakdown is empty.

Following the addition of persistent storage, it will likely be helpful for users to have the ability to select start and end dates for the data visualizations on each page.

Cumulatives / totals are giving not-a-number on our metric data. It seems to be due to the Usage API not returning totals for the last day in the extract. Not sure why not, I guess that might be a community question. Example:

{
        "day": "2024-04-23",
        "breakdown": [
            {
                "language": "java",
                "editor": "jetbrains",
                "suggestions_count": 56,
                "acceptances_count": 16,
                "lines_suggested": 99,
                "lines_accepted": 20,
                "active_users": 2
            },
            {
                "language": "kotlin",
                "editor": "jetbrains",
                "suggestions_count": 27,
                "acceptances_count": 8,
                "lines_suggested": 41,
                "lines_accepted": 7,
                "active_users": 2
            },

I'm trying to deploy copilot-metrics-viewer to run in k8s and would prefer to set the VUE_APP_GITHUB_TOKEN from an environment variable. Short of that, where is .env located in the docker image and how would i overwrite it?

The ghcr.io/github-copilot-resources/copilot-metrics-viewer:latest container image is showing quite a high number of vulnerability findings using a scanner such as Prisma Twistlock - with almost three dozen of the CVEs being rated critical.

Is there any possibility of bumping the nginx image for the production stage to 1.26?

In the latest release - v1.7 - Team metrics were made available 🎉. Take a look at the Enterprise Team and Organization Team API docs.

The GitHub Team is picked up at the moment the application is loaded via an environment variable and cannot be changed in runtime. I don't think this is ideal (this is a first version due to my low availability in the past couple weeks but I wanted to make this feature available as soon as possible).

I'm thinking about having a dropdown that lists the Teams for the Organization (or Enterprise - however no API is available to list Enterprise Teams yet) and users are able to pick the Team. Once picked, the metrics across the whole application would be refreshed. Any comments on this? Where do you think the dropdown should be added (in a new tab, menu, toolbar?) Thanks

the Logo image link is broken because there is no assets folder

Recommend emphasizing that support / questions should be submitted as an issue on this repo, not through GitHub support, either by pushing it up in the Readme or in some other way making it more prominent

hello,
Do you happen to have any update regarding the team slicing feature?
thought will be available by the end of June

As documented, the API contains metrics for the prior 28 days. It would be useful if there would be a way to configure this to support a 'local' json file containing longer-term data (e.g. we currently periodically merge the results into a larger json file with more historical data). Or, perhaps, support an external datasource where the API results are collected and persisted for a longer historical view.

In the readme the following information is provided.

VUE_APP_GITHUB_TOKEN
Specifies the GitHub Personal Access Token utilized for API requests. Generate this token with the following scopes: copilot, manage_billing:copilot, manage_billing:enterprise, read:enterprise, admin:org.

I believe that admin:org is overkill. When I did my local setup, I used the less intrusive scope read:org, and it worked just fine. If there is no functionality I'm missing, I would suggest that the README be updated to adhere to the principle of least privilege.

Whatever the data is available in the metrics result, the cumulative data is for every field is matching but not for active users.
For example: Let's take a day data result, in that breakdown field we have individual fields like suggestions, editors per active users
But the cumulative active user's summation didn't match with total active users rest of the fields to total counts are matching.

Hi,

We have encountered a problem where it is currently not possible to pass environment variables to a Docker container from outside. This seems to be due to the way the variables are integrated.

It would be much nicer if we could pass the variables through a separate environment file, such as a config.json.

Then, we could refer to the config.json when running npm serve, and in the Docker context, an environment variable or an env-file could be passed.

This would mean that no variables are required at build time.

Also, this approach would eliminate the need to constantly rebuild the container or run npm install, and it would be possible to provide one's own access token or other Vue variables at container start, without them being stored in the container during the build.

However, this would require adjusting how the variables are processed.

Thanks

Dear Team.

We have found issues with the data.

You can see the screenshot, The acceptance rate average is 33.65%, In the line chart below, the acceptance rate in the past 28 days has not exceeded 30%.

I heard some copilot users are based on github organizations. they don't have github enterprise/standalone. so such orgs are in-parrel and there is no a partent enterprise, and they do need to check usage/seat report by organizations. even such orgs are not within same enterprise, is it a popular need? thanks for your input.

          We might need to update this image once the changes have been implemented 🙏🏼

Originally posted by @martedesco in #32 (comment)

To describing my question:

Earlier i do have an organization permission. After the tokens are expired for that permission, we created the tokens for enterprise
When i used to bring the data for organization the tokens don't work there but the tokens are working with enterprise URL.

My Questions:

1. Do the enterprise metrics will include all the organizations under that access?
2. what is the distinction between enterprise members metrics and organization members?

Thanks in advance. 💯 😊 Appreciate your answer.

Once this is deployed and set up as a dashboard, does the tool automatically sync the data every day.

Hi,
I just discovered this great project.

Please consider building and hosting the docker image using GitHub Actions and the GitHub Container Registery (GHCR).

It would ease the usage of the project significantly.

The cumulative and total numbers displayed on the tiles are producing NaN on the metric data. Not sure why this is occurring. PS this isn't the same issues as this previous ticket.

Thanks

Looking at the scope required for the PAT, I am wondering why the PAT needs billingmanagement level scope. On first look, the app could/should work with "readonly" yes?

It would be nice to be able to see the product version at the bottom of the screen or somewhere else.

In the .env file, there is an option to connect with a GitHub Personal Access Token:

VUE_APP_GITHUB_TOKEN=

In large organizations, there are policies against over granting privileges to tokens for sensitive things like copilot billing and usage. Following the principle of least privilege, we are required to access sensitive data via a GitHub app.

Support for connecting through a GitHub app would allow us to use this project as is. For example:

VUE_APP_GITHUB_KEY=<name of the private key file>
VUE_APP_GITHUB_APPLICATION_ID=<application ID>
VUE_APP_GITHUB_INSTALLATION_ID=<installation ID>

A workaround is to extract the data from the GitHub Copilot Usage endpoint, and store it in a file called
src/resources/metricsExample.json and set VUE_APP_MOCKED_DATA=true. This would require additional automation, and the preferred way would be to pull real time data from the API with a GitHub App.

In order to encourage use of copilot, it would be helpful if there were a sortable table that showed all assigned users and their last (if any) access time in order to solicit users abandoning copilot for feedback.

To be clear - I'm using an "enterprise". I see that there is logic that appear to restrict the Seat Analysis view to Organizations

Hey folks,

Not clear how is possible to do the team level aggregates. I couldn't find any API that could help link the usage metrics with the teams assigned to repo's. Is there some documentation that you can point me to, on how to do it so I can give it a nudge? or it's something that we are waiting from GITHUB to implement so we can use?

Thanks,
George

It would be useful to filter metric results on users, projects, teams. I don't see a current way of doing this. Thoughts?
Thanks #

Can someone please explain how the average calculations are computed?

Looking at the values in the readme for Cumulative Number of Accepted Prompts (50734) and Cumulative Number of Suggested Prompts (199097) I would expect the average would be 50734 / 199097 * 100 or 25.48% whereas the image shows 22.19%

I ran this locally again my Enterprise and the values are 20024 and 68238 so I would expect an average of 29.34% yet the value given is 20.54%

I would like to understand where these differences are coming from so that I can make sure I have the correct values.

Clicking the column headers does not seem to re-sort the data in the associated table