Giter VIP home page Giter VIP logo

github-org-scripts's Introduction

github org helper scripts

Current minimums: python 3.7; GitHub3.py 1.3.0

Please use poetry to manage virtual environments - requirements.txt may be out of date, and does not include development dependencies.

These are some API helper scripts for sanely managing a github org. For now this is somewhat hardcoded for the mozilla org; no need for it to remain that way though. Many scripts support the --help option. That information should be more up to date than information in this document.

Credentials

Supplying credentials for execution is done by passing a PAT token as the value of the environment variable GITHUB_TOKEN (preferred) or GITHUB_PAT.

The recommended way to set GITHUB_TOKEN is via cli access to your password manager. For example, using pass:

GITHUB_TOKEN=$(pass show myPAT) script args

Jupyter Notebooks

Docker Images

Our Jupyter Notebooks have a farely simple environment as regards dependencies. The recommended way to deal with this is by using a docker container.

The Makefile contains targets for building and running the docker images. Invoke make without arguments to see those targets

  • NOTE: the docker image allows credentials to be supplied via sops. The environment variable "SECOPS_SOPS_PATH" must be set appropriately.

When started, the docker container will serve notebooks from the notebooks/ directory, but they will be available at the top level. Current notebooks include:

  • User Search.ipynb -- Given a set of possible GitHub logins, determine if they might have any permissions in various organizations. Links are provided for hits, so easy to examine more closely.

    N.B.: Both this script and the GitHub search interface make assumptions. It is your responsibility to ensure any possible match is a valid match.

    There is now a section which will search for usernames in any non-documentation source file. The intent is to spot cases where app, login, or other permissions may have been granted via that file. Since such authorization usage is adhoc, there are likely to be many false positives. (However teams may choose to use the list for "cleanup" of unmaintained documents.) Typically, the user will want to supply both ldap and GitHub logins to be the search targets.

Scripts

Scripts should now work with Python 3. Please open issues for any problems you encounter.

auditlog.py

Download audit log for $ORG via headless firefox via selenium (geckodriver must be installed). Credentials as environment variables, and 2FA token passed as input when requested.

contributing.py

Analyze all the "sources" repositories (i.e., those that aren't forks) in a github org and list the repositories that do NOT have a CONTRIBUTING file.

get_active_hooks.py

Find all hooks configured for an organization -- see --help for details

get_org_info.py

Output basic info about an org, more if you have permissions. See --help for details

manage_invitations.py

Cancel all org & repository invitations older than a specified age (default 2 weeks). See --help for details.

team_update.py

Update administrative teams so they can be used for the new GitHub discussion feature. Use the --help option for more information.

hooks.py

Analyzes a list of audit log export files (from the JS script) for hook/service creation/deletion and provides a summary. Use it to show commonly used apps/services/webhooks across the org.

old_repos.py

Generate a list of empty (should be deleted) repositories as well as untouched repos (might need to be archived).

BUGS

  • Some of these scripts are no longer relevent.

License

This code is free software and licensed under an MPL-2.0 license. © 2015-2021 Fred Wenzel and others. For more information read the file LICENSE.

github-org-scripts's People

Contributors

cknowles-moz avatar dependabot[bot] avatar fwenzel avatar g-k avatar glogiotatidis avatar groovecoder avatar hwine avatar moz-hwine avatar mozilla-github-standards avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

github-org-scripts's Issues

Convert to MPL?

Not sure if there's a reason early work in this repo was BSD. Would seem to make more sense to have under MPL.

@fwenzel do you know the original reason? Or have objection to switching?

LFS usage script

Description from @hwine via email:

  • iterate over all repositories using api.github.com
  • for each repository, attempt to retrieve the raw .gitattributes file
    • if it does not exist, that repo does not have LFS
    • if it does exist, examine contents to see if LFS attributes are there (this is the part I don't have figured out, but appears to be well documented in the LFS project)
  • once it's known a given repo has LFS attributes:
    • clone the repo (a shallow clone may suffice)
    • execute git lfs ls-files <= gives counts & size

Team management scripts need exclusion list

In general, the usage of the team management scripts is to add persons to teams so they can be contacted.

This means that "service accounts" should not be added. Which means that we need an exclusion list somewhere....

To Do:

  • figure out where to store per organization
  • update code to honor those lists

Wiki changes

FYI: The following changes were made to this repository's wiki:

These were made as the result of a recent automated defacement of publically writeable wikis.

Update instructions re repo2docker

For some reason (I found it once, but have since forgotten), repo2docker needs chardet installed as well. I installed repo2docker with pipx, so only had to manually install chardet once.

  • Update docs to specify checking for chardet and install if needed. Docs lean towards folks installing in the virtualenv (which makes sense)
  • also catch default branch is 'main'
  • be more explicit about env variable names
  • poetry instructions explicitly
  • update docs for Makefile target changes

Document credential handling

Add to the readme that credentials are expected to be found (by the GitHub3.py library) in .credentials in the project root (or script directory - not sure which is technically correct - it's in the .gitignore file).

Line 1: GitHub login to use
Line 2: API token in plain text

Rest of file ignored

admins.py is broken

Must have been a GitHub API change or something. admins.py is certainly not showing admins without 2FA is intended. It seems like it might be showing all members without 2FA?

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

  1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
  2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please see Mozilla-GitHub-Standards or email [email protected].

(Message COC001)

[User Search.ipynb] No guidance provided for when manual searching is needed

For folks in certain departments, it would be unusual for them not to have a GitHub login. At the same time, there are many exceptions, e.g. when developers only work on hg.m.o repositories. And many "that's okay" such as folks in non technical roles.

Currently, making the above distinction is left entirely to the human. Given the typical use case is to run this for a large number of folks (e.g. intern departure day), this leaves too much room for human error.

Enhancing the output to make a guess as to whether or not a GitHub login should have been found would be awesome.

intent to upgrade github3.py library

The newer (1.0.0a2) develop branch versions of the github3.py package have a number of nice additions to support the modern Organization features.
However, there are a number of breaking changes that will require rework of some existing scripts.
Because of the compatibility issues, and that the upstream copy does not yet have a release date, I'll do this work on a new branch, which will include the github3.py code as a git subtree (manual method).

Any objections or concerns? Please post in this issue.

Remove dependency on repo2docker

See #81 for problems using repo2docker. In addition, it is very slow to build.

Our needs are simple, we don't need the features repo2docker provides.

Invitations to be a contributor are not managed

Invitations to be an organization member can be cancelled, if not accepted, via ./manage_invitations.py.

It is not performing the same task for invitations to be a contributor. While these invitations are specific to a repository, they are displayed in the UI at https://github.com/orgs/<organization>/pending_collaborators.

While similar in concept, contributor invitations have a different response format.

Create tool to auto add issue & PR for Mozilla code of conduct

A goal for 2019 is to have every Mozilla repository on GitHub contain a standard code of conduct. (This applies to all Mozilla orgs, not just 'mozilla'.)

The current plan is to have tools that will:

  • determine if a repository is compliant
  • open an issue to bring into compliance
  • open a PR to satisfy that issue (stretch goal)
  • if the repo has a contributing.md file, generate a PR to prepend a local reference to the code of conduct file (stretch goal)

This is the master tracker issue for the work.

Check for "owned resources" when revoking access

before revoking access for a login, it'd be a nice thing to do to look for any credentials that may be tied to that login.

In many cases, that is not observable (e.g. PAT in automation somewhere), but at least the following could be looked for:

  • deploy keys authorized by login

[User Search.ipynb] Update to work with GitHub3.py version 1.3.0

Commit 61eb739 updated GitHub3.py to version 1.3.0, which broke User Search.ipynb. The notebook scripts should be updated.

Workaround until this is fixed:

  1. activate venv
  2. pip install -U git+https://github.com/hwine/github3.py.git@invitations
  3. restart the kernel in the notebook
  4. restore the current environment with pip install -r requirements.txt

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.