Giter VIP home page Giter VIP logo

git-vuln-finder's Introduction

git-vuln-finder

git-vuln-finder logo

Workflow

Finding potential software vulnerabilities from git commit messages. The output format is a JSON with the associated commit which could contain a fix regarding a software vulnerability. The search is based on a set of regular expressions against the commit messages only. If CVE IDs are present, those are added automatically in the output. The input can be any git repositories or a GH archive source.

Requirements

  • jq (sudo apt install jq)
  • Python poetry

Installation

Use it as a library

git-vuln-finder can be install with poetry. If you don't have poetry installed, you can do the following curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python.

$ poetry install
$ poetry shell
$ git-vuln-finder -h

You can also use pip. Then just import it:

Python 3.8.0 (default, Dec 11 2019, 21:43:13)
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from git_vuln_finder import find
>>> all_potential_vulnerabilities, all_cve_found, found = find("~/git/curl")

>>> [commit for commit, summary in all_potential_vulnerabilities.items() if summary['state'] == 'cve-assigned']
['9069838b30fb3b48af0123e39f664cea683254a5', 'facb0e4662415b5f28163e853dc6742ac5fafb3d',
... snap ...
 '8a75dbeb2305297640453029b7905ef51b87e8dd', '1dc43de0dccc2ea7da6dddb7b98f8d7dcf323914', '192c4f788d48f82c03e9cef40013f34370e90737', '2eb8dcf26cb37f09cffe26909a646e702dbcab66', 'fa1ae0abcde5df8d0b3283299e3f246bedf7692c', 'c11c30a8c8d727dcf5634fa0cc6ee0b4b77ddc3d', '75ca568fa1c19de4c5358fed246686de8467c238', 'a20daf90e358c1476a325ea665d533f7a27e3364', '042cc1f69ec0878f542667cb684378869f859911']

 >>> print(json.dumps(all_potential_vulnerabilities['9069838b30fb3b48af0123e39f664cea683254a5'], sort_keys=True, indent=4, separators=(",", ": ")))
 {
     "author": "Daniel Stenberg",
     "author-email": "[email protected]",
     "authored_date": 1567544372,
     "branches": [
         "master"
     ],
     "commit-id": "9069838b30fb3b48af0123e39f664cea683254a5",
     "committed_date": 1568009674,
     "cve": [
         "CVE-2019-5481",
         "CVE-2019-5481"
     ],
     "language": "en",
     "message": "security:read_data fix bad realloc()\n\n... that could end up a double-free\n\nCVE-2019-5481\nBug: https://curl.haxx.se/docs/CVE-2019-5481.html\n",
     "origin": "https://github.com/curl/curl.git",
     "origin-github-api": "https://api.github.com/repos///github.com/curl/curl/commits/9069838b30fb3b48af0123e39f664cea683254a5",
     "pattern-matches": [
         "double-free"
     ],
     "pattern-selected": "(?i)(double[-| ]free|buffer overflow|double free|race[-| ]condition)",
     "state": "cve-assigned",
     "stats": {
         "deletions": 4,
         "files": 1,
         "insertions": 2,
         "lines": 6
     },
     "summary": "security:read_data fix bad realloc()",
     "tags": []
 }

Use it as a command line tool

$ git clone https://github.com/cve-search/git-vuln-finder.git
$ cd https://github.com/cve-search/git-vuln-finder.git
$ pip install .
$ git-vuln-finder --help

You can also use pip. pipx installs scripts (system wide available) provided by Python packages into separate virtualenvs to shield them from your system and each other.

Usage

usage: git-vuln-finder [-h] [-v] [-r R] [-o O] [-s S] [-p P] [-c] [-t] [-gh GH]

Finding potential software vulnerabilities from git commit messages.

optional arguments:
  -h, --help  show this help message and exit
  -v          increase output verbosity
  -r R        git repository to analyse
  -o O        Output format: [json]
  -s S        State of the commit found
  -p P        Matching pattern to use: [vulnpatterns, cryptopatterns, cpatterns] - the pattern 'all' is used to match all the patterns at once.
  -c          output only a list of the CVE pattern found in commit messages (disable by default)
  -t          Include tags matching a specific commit
  -gh GH      special option for gharchive, pass a file containing a PushEvent in JSON format

More info: https://github.com/cve-search/git-vuln-finder

Patterns

git-vuln-finder comes with 3 default patterns which can be selected to find the potential vulnerabilities described in the commit messages such as:

  • vulnpatterns is a generic vulnerability pattern especially targeting web application and generic security commit message. Based on an academic paper.
  • cryptopatterns is a vulnerability pattern for cryptographic errors mentioned in commit messages.
  • cpatterns is a set of standard vulnerability patterns see for C/C++-like languages.

A sample partial output from Curl git repository

$ git-vuln-finder -r ~/git/curl | jq .
...
 "6df916d751e72fc9a1febc07bb59c4ddd886c043": {
    "message": "loadlibrary: Only load system DLLs from the system directory\n\nInspiration provided by: Daniel Stenberg and Ray Satiro\n\nBug: https://curl.haxx.se/docs/adv_20160530.html\n\nRef: Windows DLL hijacking with curl, CVE-2016-4802\n",
    "language": "en",
    "commit-id": "6df916d751e72fc9a1febc07bb59c4ddd886c043",
    "summary": "loadlibrary: Only load system DLLs from the system directory",
    "stats": {
      "insertions": 180,
      "deletions": 8,
      "lines": 188,
      "files": 7
    },
    "author": "Steve Holme",
    "author-email": "[email protected]",
    "authored_date": 1464555460,
    "committed_date": 1464588867,
    "branches": [
      "master"
    ],
    "pattern-selected": "(?i)(denial of service |\bXXE\b|remote code execution|\bopen redirect|OSVDB|\bvuln|\bCVE\b |\bXSS\b|\bReDoS\b|\bNVD\b|malicious|x−frame−options|attack|cross site |exploit|malicious|directory traversal |\bRCE\b|\bdos\b|\bXSRF \b|\bXSS\b|clickjack|session.fixation|hijack|\badvisory|\binsecure |security |\bcross−origin\b|unauthori[z|s]ed |infinite loop)",
    "pattern-matches": [
      "hijack"
    ],
    "origin": "[email protected]:curl/curl.git",
    "origin-github-api": "https://api.github.com/repos/curl/curl/commits/6df916d751e72fc9a1febc07bb59c4ddd886c043",
    "tags": [],
    "cve": [
      "CVE-2016-4802"
    ],
    "state": "cve-assigned"
  },
  "c2b3f264cb5210f82bdc84a3b89250a611b68dd3": {
    "message": "CONNECT_ONLY: don't close connection on GSS 401/407 reponses\n\nPreviously, connections were closed immediately before the user had a\nchance to extract the socket when the proxy required Negotiate\nauthentication.\n\nThis regression was brought in with the security fix in commit\n79b9d5f1a42578f\n\nCloses #655\n",
    "language": "en",
    "commit-id": "c2b3f264cb5210f82bdc84a3b89250a611b68dd3",
    "summary": "CONNECT_ONLY: don't close connection on GSS 401/407 reponses",
    "stats": {
      "insertions": 4,
      "deletions": 2,
      "lines": 6,
      "files": 1
    },
    "author": "Marcel Raad",
    "author-email": "[email protected]",
    "authored_date": 1455523116,
    "committed_date": 1461704516,
    "branches": [
      "master"
    ],
    "pattern-selected": "(?i)(denial of service |\bXXE\b|remote code execution|\bopen redirect|OSVDB|\bvuln|\bCVE\b |\bXSS\b|\bReDoS\b|\bNVD\b|malicious|x−frame−options|attack|cross site |exploit|malicious|directory traversal |\bRCE\b|\bdos\b|\bXSRF \b|\bXSS\b|clickjack|session.fixation|hijack|\badvisory|\binsecure |security |\bcross−origin\b|unauthori[z|s]ed |infinite loop)",
    "pattern-matches": [
      "security "
    ],
    "origin": "[email protected]:curl/curl.git",
    "origin-github-api": "https://api.github.com/repos/curl/curl/commits/c2b3f264cb5210f82bdc84a3b89250a611b68dd3",
    "tags": [],
    "state": "under-review"
  },
...
  • Extracting CVE id(s) from git messages
  "98d132cf6a879faf0147aa83ea0c07ff326260ed": {
    "message": "Add a macro for testing assertion in both debug and production builds\n\nIf we have an assert then in a debug build we want an abort() to occur.\nIn a production build we wan
t the function to return an error.\n\nThis introduces a new macro to assist with that. The idea is to replace\nexisting use of OPENSSL_assert() with this new macro. The problem with\nOPENSSL
_assert() is that it aborts() on an assertion failure in both debug\nand production builds. It should never be a library's decision to abort a\nprocess (we don't get to decide when to kill t
he life support machine or\nthe nuclear reactor control system). Additionally if an attacker can\ncause a reachable assert to be hit then this can be a source of DoS attacks\ne.g. see CVE-20
17-3733, CVE-2015-0293, CVE-2011-4577 and CVE-2002-1568.\n\nReviewed-by: Tim Hudson <[email protected]>\n(Merged from https://github.com/openssl/openssl/pull/3496)",
    "commit-id": "98d132cf6a879faf0147aa83ea0c07ff326260ed",
    "summary": "Add a macro for testing assertion in both debug and production builds",
    "stats": {
      "insertions": 18,
      "deletions": 0,
      "lines": 18,
      "files": 1
    },
    "author": "Matt Caswell",
    "author-email": "[email protected]",
    "authored_date": 1495182637,
    "committed_date": 1495457671,
    "branches": [
      "master"
    ],
    "pattern-selected": "(?i)(denial of service |\bXXE\b|remote code execution|\bopen redirect|OSVDB|\bvuln|\bCVE\b |\bXSS\b|\bReDoS\b|\bNVD\b|malicious|x−frame−options|attack|cross site |ex
ploit|malicious|directory traversal |\bRCE\b|\bdos\b|\bXSRF \b|\bXSS\b|clickjack|session.fixation|hijack|\badvisory|\binsecure |security |\bcross−origin\b|unauthori[z|s]ed |infinite loop)",
    "pattern-matches": [
      "attack"
    ],
    "cve": [
      "CVE-2017-3733",
      "CVE-2015-0293",
      "CVE-2011-4577",
      "CVE-2002-1568"
    ],
    "state": "cve-assigned"
  }

Usage for the special gharchive option

$ git-vuln-finder -gh ../tests/gharchive_test.json

the value for the gh parameters need to be a json file, containing an array of each PushEvent you want to test.

[
   {
    "id": "19351512310",
    "type": "PushEvent",
    "actor": {
      "id": 32466128,
      "login": "DavidCruciani",
      "display_login": "DavidCruciani",
      "gravatar_id": "",
      "url": "https://api.github.com/users/DavidCruciani",
      "avatar_url": "https://avatars.githubusercontent.com/u/32466128?"
    },
    "repo": {
      "id": 424660123,
      "name": "ail-project/ail-feeder-gharchive",
      "url": "https://api.github.com/repos/ail-project/ail-feeder-gharchive"
    },
    "payload": {
      "push_id": 8628652926,
      "size": 1,
      "distinct_size": 1,
      "ref": "refs/heads/main",
      "head": "910ed71a2819546a3f3bcce1ebb9e3984a8c8d86",
      "before": "40a9ef5dc6b2add5184a0a58401bfe9058faa8df",
      "commits": [
        {
          "sha": "910ed71a2819546a3f3bcce1ebb9e3984a8c8d86",
          "author": {
            "email": "[email protected]",
            "name": "David Cruciani"
          },
          "message": "chg: [feeder] case sensitive",
          "distinct": true,
          "url": "https://api.github.com/repos/ail-project/ail-feeder-gharchive/commits/910ed71a2819546a3f3bcce1ebb9e3984a8c8d86"
        }
      ]
    },
    "public": true,
    "created_at": "2021-12-15T16:06:43Z",
    "org": {
      "id": 62389074,
      "login": "ail-project",
      "gravatar_id": "",
      "url": "https://api.github.com/orgs/ail-project",
      "avatar_url": "https://avatars.githubusercontent.com/u/62389074?"
    }
  }
]

Usage for import

If the goal is to import the module to use it, the method to call is find_event

from git_vuln_finder import find_event

for element in event:
    for i in range(0,len(element["payload"]["commits"])):
        all_potential_vulnerabilities, all_cve_found, found = find_event(element["payload"]["commits"][i], element)

Output with gharchive option

{
    "repo_name": "LeandroFChaves/gerenciador-alunos",
    "message": "[UI] - Ajustes no css da aplica\u00e7\u00e3o\n\n- Adicionado padding para a exibi\u00e7\u00e3o do conte\u00fado das p\u00e1ginas;\n- Alinhado os bot\u00f5es de a\u00e7\u00f5es dos forms a direita da table",
    "language": "pt",
    "commit-id": "73a1c68b520853198eaac199a41d141ee96dc64d",
    "author": "LeandroFChaves",
    "author-email": "[email protected]",
    "authored_date": "2021-10-01T03:00:07Z",
    "branches": "refs/heads/master",
    "pattern-selected": "(?i)(denial of service|\\bXXE\\b|remote code execution|\\bopen redirect|OSVDB|\\bvuln|\\bCVE\\b|\\bXSS\\b|\\bReDoS\\b|\\bNVD\\b|malicious|x\u2212frame\u2212options|attack|cross site|exploit|malicious|directory traversal|\\bRCE\\b|\\bdos\\b|\\bXSRF \\b|\\bXSS\\b|clickjack|session.fixation|hijack|\\badvisory|\\binsecure|security|\\bcross\u2212origin\\b|unauthori[z|s]ed|infinite loop)",
    "pattern-matches": [
        "dos"
    ],
    "origin-github-api": "https://api.github.com/repos/LeandroFChaves/gerenciador-alunos/commits/73a1c68b520853198eaac199a41d141ee96dc64d",
    "state": "under-review"
}

Running the tests

$ pytest

Warning

git-vuln-finder is an automatic tool that detects potential vulnerabilities in commit messages based on specific keywords. If you intend to use the discovered vulnerability information, such as CVEs registration, please review the discovered vulnerabilities with the authors of the software before proceeding.

License and author(s)

This software is free software and licensed under the AGPL version 3.

Acknowledgment

  • Thanks to Jean-Louis Huynen for the discussions about the crypto vulnerability patterns.
  • Thanks to Sebastien Tricaud for the discussions regarding native language, commit messages and external patterns.
  • Thanks to Cedric Bonhomme to make git-vuln-finder a Python library, add tests and improve the overall installation process.
  • Thanks to David Cruciani for the support of gharchive.

Contributing

We welcome contributions for the software and especially additional vulnerability patterns. Every contributors will be added in the AUTHORS file and collectively own this open source software. The contributors acknowledge the Developer Certificate of Origin.

References

git-vuln-finder's People

Contributors

adulau avatar cedricbonhomme avatar cydave avatar davidcruciani avatar dependabot[bot] avatar gallypette avatar tdunlap607 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

git-vuln-finder's Issues

Tool never finds any CVE in commit - it doesnt find the commits.

Tried testing on repos, inc curl - the tool alays shows "found in 0 commits"

git-vuln-finder -r curl/.git/ | jq .
['master']
0 CVE referenced found in commit(s)
Total potential vulnerability found in 0 commit(s)

syntax in readme doesnt work

the 0 commits part shows its not actually testing the repo

Issue on MacOS Catalina using zsh terminal

Unhandled LangDetectException

Some commit-messages do not contain any valid words (from known languages), which leads to Exception from langdetect-package. This exception isn't handled, causing the program to stop. It would be nice, if the finder could simply catch and set language to None, when such kind of Exception is raised.

For example:

A try to parse https://github.com/jenkinsci/script-security-plugin will always fail, because it's repository has a commit named like this:

https //github.com/jenkinsci/script-security-plugin/pull/348#discussion_r634333189

'vulnerablity' instead of 'vulnerability'

The following file "git-vuln-finder/git_vuln_finder/patterns/en/medium/crypto" has the keyword "vulnerablity" but no "vulnerability". Is this intentional or is it a typo?

Preparing metadata (pyproject.toml) ... error

I have some problem when install this tools like this :

Processing /Users/mac/Documents/Development/My Project/Python/git-vuln-finder
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [17 lines of output]
      Traceback (most recent call last):
        File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/_x/x4pdztk914j_jxcfzgd7x8xr0000gn/T/pip-build-env-s73_0r3_/overlay/lib/python3.11/site-packages/poetry/core/masonry/api.py", line 42, in prepare_metadata_for_build_wheel
          poetry = Factory().create_poetry(Path(".").resolve(), with_groups=False)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/_x/x4pdztk914j_jxcfzgd7x8xr0000gn/T/pip-build-env-s73_0r3_/overlay/lib/python3.11/site-packages/poetry/core/factory.py", line 58, in create_poetry
          raise RuntimeError("The Poetry configuration is invalid:\n" + message)
      RuntimeError: The Poetry configuration is invalid:
        - data.documentation must be uri
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Please help me, how to solve ? thanks in advance.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.