A set of tools to extract and analyze the number of software contributions and their submitter. This is a strict interpretation of "contribution". Other statistics (lile those from the Linux Foundation), analyse all interactions with a project (PR, comment, issue creation, review).
- Retrieve data since the last time the script ran with
./collect-missing-data.sh. - Update/create the consolidated data file with
./consolidate-data.sh. - Update/create global summary file (nbr submitters and submissions per month) with
./submission-submitter-report.sh.
All the above operations can be performed with update-stats.sh
Following scripts are available:
-
check-prerequisites.shchecks whether all required programs are available on the system -
extract-monthly-submissions.shextracts the monthly data from GitHub and stores it in ,/data/ directory as a CSV file. It also generates a list with the number of PR for each submitter in the given month. -
consolidate-data.shtakes all the available monthly data and creates a single data file,consolidated_data/submissions.csv. If a data file already exists, it is backuped. -
collect-missing-data.shwill extract all the monthly data files since JAN-2020. If the output already exists, it will skip that particular month. -
submission-submitter-report.shuses the existing monthly data to generate a summary CSV with the number of submissions and the number of submitters. The resulting output is stored inconsolidated_data/summary_counts.csv -
update-stats.shis the script that performs the necessary update operation in sequence
| File name | Comment | produced by |
|---|---|---|
data/submissions-YYYY-MM.csv |
List of PRs created in a given month | extract-monthly-submissions.sh |
data/pr_per_submitter-YYYY-MM.csv |
Nbr of PRs submitted by a user for a given month | extract-monthly-submissions.sh |
data/comments_YYYY-MM.csv |
List of comments created in a given month | extract-monthly-submissions.sh |
data/comments_per__commenter-YYYY-MM.csv |
Nbr of comments made by a user for a given month | extract-monthly-submissions.sh |
consolidated_data/submissions.csv |
All extracted submissions (since Jan 2020) | consolidate-data.sh submissions |
consolidated_data/submissions_overview.csv |
Global submissions pivot table (user/month -> nbr prs) | consolidate-data.sh submissions |
consolidated_data/top_submissions.csv |
35 top submitters over the last 12 month | consolidate-data.sh submissions |
consolidated_data/top_submissions_evolution.csv |
New or churned top submitters (compared to 3 months before) | consolidate-data.sh submissions |
consolidated_data/comments.csv |
All extracted comments (since Jan 2020) | consolidate-data.sh comments |
consolidated_data/submissions_overview.csv |
Global comments pivot table (user/month -> nbr comments) | consolidate-data.sh comments |
consolidated_data/top_submissions.csv |
35 Top commenters over the last 12 months | consolidate-data.sh comments |
consolidated_data/top_submissions_evolution.csv |
New or churned top commenters (compared to 3 months before) | consolidate-data.sh comments |
Prerequisites are checked with check-prerequisite.sh.
This is the list of executables that have to be installed in order for the automation to work.
gh: the Github command line utilityjq: Json query tooldatamash: data manipulation tool (CSV pivots)jenkins-contribution-aggregator: extracts the to submitters or commenters from the global pivot tablesjenkins-contribution-extractor: various extraction and jenkins data handling toolsgdate: GNU date manipulation tool for Mac OS (part ofcoreutils, installable with brew)
Not to self: to generate the mermaid graphic by hand docker run -i -t --rm -v "$PWD:/data" jmmeessen/render-md-mermaid:v2
diagram source
This details block is collapsed by default when viewed in GitHub. This hides the mermaid graph definition, while the rendered image linked above is shown. The details tag has to follow the image tag. (newlines allowed)flowchart TD
start1(("`Start
(others)
`"))
start2(("`Start
(jenkins)
`"))
extract_end((End))
%% Processes
A[[update-benchmark-stats.sh]]
B[[update-stats.sh]]
C[[collect-missing-data.sh]]
D[[consolidate-data.sh submissions]]
E[[consolidate-data.sh comments]]
F[[submission-submitter-report.sh]]
G[[comment-commenter-report.sh]]
extracData[[extract-monthly-submissions.sh]]
get_submitters{{"jenkins-contribution-extractor get submitters {org}"}}
get_commenters{{"jenkins-contribution-extractor get commenters"}}
top_extract{{jenkins-contribution-aggregator </br> extract}}
top_compare{{jenkins-contribution-aggregator </br>compare}}
%% data files
submission_month[(submission_YYMM.csv)]
monththlyPivot_submit[(pr_per_submitter.csv)]
comments_month[(comments_YYMM.csv)]
monththlyPivot_comment[(comments_per_</br>_commenter.csv)]
global_submissions[(submissions.csv)]
global_submissionsOverview[(submissions_overview.csv)]
top_submission[(top_submissions.csv)]
top_submission_evol[(top_submissions_evolution.csv)]
global_comments[(comments.csv)]
global_commentsOverview[(comments_overview.csv)]
%% legend
legend_app[[Application </br>or script]]
legend_sub{{sub routine}}
legend_data[(data file)]
legend_app --> legend_sub -.-> legend_data
%% pivot processes
monthlypivot_subm{{pivot monthly data}}
monthlypivot_comment{{pivot monthly data}}
subm_overview_pivot{{pivot}}
comment_overview_pivot{{pivot}}
%% flow
start1 --> A -- loops through org --> B
start2 --> B
B --> C -- monthly data missing ? --> extracData --> get_submitters
get_submitters -.-> submission_month --> monthlypivot_subm -.-> monththlyPivot_submit --> extract_end --> C
submission_month --> get_commenters -.-> comments_month --> monthlypivot_comment -.-> monththlyPivot_comment --> extract_end
B --> D --> global_submissions
global_submissions --> subm_overview_pivot -.-> global_submissionsOverview
global_submissions --> top_extract --> top_submission
global_submissions --> top_compare --> top_submission_evol
B --> E --> global_comments --> comment_overview_pivot -.-> global_commentsOverview
B --> F
B --> G
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "github-actions[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent