americanexpress / earlybird Goto Github PK

EarlyBird is a sensitive data detection tool capable of scanning source code repositories for clear text password violations, PII, outdated cryptography methods, key files and more.

License: Apache License 2.0

Batchfile 0.72% Shell 1.18% Go 84.46% Rich Text Format 13.64%

earlybird's Introduction

EarlyBird is a sensitive data detection tool capable of scanning source code repositories for clear text password violations, PII, outdated cryptography methods, key files and more. It can be used to scan remote git repositories, local files or directories or as a pre-commit step.

Installation

Linux & Mac

Running the build.sh script will produce a binary for each OS, while the install.sh script will install Earlybird on your system. This will create a .go-earlybird directory in your home directory with all the configuration files. Finally installing go-earlybird as an executable in /usr/local/bin/.

./build.sh && ./install.sh

Windows

Running build.bat will produce your binaries while the install.bat script will create a 'go-earlybird' directory in C:\Users\[my user]\App Data\, and copy the required configurations there. This script will also install go-earlybird.exe as an executable in the App Data directory (which should be in your path).

build.bat && install.bat

Usage

To launch a basic EarlyBird scan against a directory:

$ go-earlybird --path=/path/to/directory

$ go-earlybird.exe --path=C:\path\to\directory

or to scan a remote git repo:

$ go-earlybird --git=https://github.com/americanexpress/earlybird

Click here for Detailed Usage instructions.

Documentation

Why Are We Doing This?

The MITRE Corporation provides a catalog of Common Weakness Enumerations (CWE), documenting issues that should be avoided. Some of the relevant CWEs that are handled by the use of EarlyBird include:

Contributing

We welcome your interest in the American Express Open Source Community on Github. Any Contributor to any Open Source Project managed by the American Express Open Source Community must accept and sign an Agreement indicating agreement to the terms below. Except for the rights granted in this Agreement to American Express and to recipients of software distributed by American Express, You reserve all right, title, and interest, if any, in and to your contributions. Please fill out the Agreement.

License

Any contributions made under this project will be governed by the Apache License 2.0.

Code of Conduct

This project adheres to the American Express Community Guidelines. By participating, you are expected to honor these guidelines.

earlybird's People

Contributors

Stargazers

Watchers

earlybird's Issues

pkg/utils: GetBBProject should hoist out the static string built regex as a global and invoke panics not os.Exit(1)

From Orijtech Cyber's audits of earlybird, we noticed that this code could be hoisted out and made a global given that the regular expression is constructed from a constant/static string per

earlybird/pkg/utils/utils.go

Lines 162 to 163 in 69c7517

 func GetBBProject(bbURL string) (project string) { 

 re := regexp.MustCompile(`(?:projects/)([^/]*)`)

to be made

diff --git a/pkg/utils/utils.go b/pkg/utils/utils.go
index 18eb9ad..87dc4b3 100644
--- a/pkg/utils/utils.go
+++ b/pkg/utils/utils.go
@@ -158,10 +158,11 @@ func GetGitRepo(gitURL string) (repository string) {
 	return repository
 }
 
+var reProject = regexp.MustCompile(`(?:projects/)([^/]*)`)
+
 // GetBBProject Parse project name from bitbucket URL
 func GetBBProject(bbURL string) (project string) {
-	re := regexp.MustCompile(`(?:projects/)([^/]*)`)
-	results := re.FindStringSubmatch(bbURL) // Match second capture group, 1 = project/XXX, 2 = XXX
+	results := reProject.FindStringSubmatch(bbURL) // Match second capture group, 1 = project/XXX, 2 = XXX
 	if len(results) < 1 {
 		log.Println("Failed To Get BB Project from URL:", bbURL)
 		os.Exit(1)

Benefits

Please read more at https://cyber.orijtech.com/scsec/cosmos-go-coding-guide#construct-regexps-once and see a real-world benchmark result

$ benchstat before.txt after.txt 
name              old time/op    new time/op    delta
RegexpCompiled-8    3.00µs ±20%    1.84µs ± 1%  -38.75%  (p=0.000 n=10+9)

name              old alloc/op   new alloc/op   delta
RegexpCompiled-8    1.13kB ± 0%    0.25kB ± 0%  -77.94%  (p=0.000 n=10+10)

name              old allocs/op  new allocs/op  delta
RegexpCompiled-8      16.0 ± 0%       6.0 ± 0%  -62.50%  (p=0.000 n=10+10)

Future proofing

Please install into your CI/CD workflows, our tool staticmajor https://github.com/marketplace/actions/staticmajor-analyzer which will report such issues beforehand.

/cc @elias-orijtech @madflojo

Configuration option to hide value from Console/Log

Earlybird currently displays the Line number and Value as part of the output.

Considering the Values have sensitive data and most integrations via CICD ship these logs to systems like DataDog and Splunk, I am proposing a config in earlybird.json to control this. This is in align with OWAP Logging - Data to Exclude

"show_value_in_result": false
The default should be "no value, " allowing users to override this.

EarlyBird Version: 3.16.0
Operating System: macOS 14.0

Uplift go.mod to v4 for stable version release.

Currently pkg.go.dev show version which not great for stable build. Let's make changes to reflect the version in go.mod and update the path in the source code.

fix code and performance audit findings

Hello American Express, from Orijtech Cyber we performed an audit of your software earlybird security secrets scanner and a great contribution to improving supply chain security generally, thank you! Please find our report at https://cyber.orijtech.com/scsec/amex-earlybird-v1

/cc @madflojo @elias-orijtech @jhusdero @willpoint

Earlybird tests

Hello,
Didn't know where to ask this question so I raised this issue.
I tried earlybird on the following poor, test, C source code :

#include<stdio.h>
#include<string.h>

int main(void) {
    char enteredPass[30];
    char password[30]="MyPassw0rd";
    printf("Enter Password:\n");
    scanf("%s", enteredPass);
    if (strcmp(enteredPass, password) == 0) {
        printf("%s is my Password!\nOops\n", password);
        return 0;
    } else {
        printf("You didn't found it!\n");
        return -1;
    }
}

and nothing is detected by earlybird.

I got :
1 files scanned in 2.048829ms
2021/10/08 11:42:22
144 rules observed
***** Total issues found *****
0 TOTAL ISSUES

How is this possible?

This is almost exactly what is described as C example in CWE-798.
Thanks for the help.

Configuration in default path is mandatory to launch the tool

Even if configuration parameter is set in command line the default configuration path is still trying to load.
And if this path is not present earlybird fails to launch with error:
"Failed to load Earlybird configopen /var/lib/jenkins/.go-earlybird/earlybird.json: no such file or directory"

Expected:
If configuration path is provided in command line, the default configuration path should not be checked.

Version: 1.24.6

Scan is randomly failing and display confidence config is not working as needed

If we pass --fail-severity and --display-confidence config, then results are not what we are expecting.

Actual result

git:main ✗ ⚑  $ go run go-earlybird.go --fail-severity high --fail-confidence medium        --display-severity high         --display-confidence medium --path=/Users/xyz/sample --enable=password-secret

2023/02/14 16:23:19 Go-EarlyBird version:  dev
Severity Fail threshold (at or above):  high
Confidence Fail threshold (at or above):  medium
Severity Display threshold (at or above):  high
Confidence Display threshold (at or above):  medium
Max file size to scan:  10240000  bytes
2023/02/14 16:23:19 loading module:  password-secret
2023/02/14 16:23:19 Scanning directory:  /Users/xyz/sample
Finding # 1:
        Code #: 3026
        Filename: /Users/xyz/sample/te.js
        Caption: Potentially Weak Password
        Category: password-secret
        Line #: 7
        Value: secret = "my123TestPassword"
        Severity: high
        Confidence: medium
        Labels: weak password
        Associated CWEs: CWE-798/CWE-312/CWE-257/CWE-259

Finding # 2:
        Code #: 3001
        Filename:  /Users/xyz/sample/te.js
        Caption: Potentially Weak Password
        Category: password-secret
        Line #: 1
        Value: password = "Prvdfgfsgd"
        Severity: high
        Confidence: medium
        Labels: weak password
        Associated CWEs: CWE-798/CWE-312/CWE-257/CWE-259

        ***** Total issues found *****
            2 Potentially Weak Password
            2 TOTAL ISSUES
2023/02/14 16:23:19 
1 files scanned in 1.867156ms
2023/02/14 16:23:19 
33 rules observed

expected result

2023/02/14 16:23:19 Go-EarlyBird version:  dev
Severity Fail threshold (at or above):  high
Confidence Fail threshold (at or above):  medium
Severity Display threshold (at or above):  high
Confidence Display threshold (at or above):  medium
Max file size to scan:  10240000  bytes
2023/02/14 16:23:19 loading module:  password-secret
2023/02/14 16:23:19 Scanning directory:  /Users/xyz/sample
Finding # 1:
        Code #: 3026
        Filename: /Users/xyz/sample/te.js
        Caption: Potentially Weak Password
        Category: password-secret
        Line #: 7
        Value: secret = "my123TestPassword"
        Severity: high
        Confidence: medium
        Labels: weak password
        Associated CWEs: CWE-798/CWE-312/CWE-257/CWE-259

Finding # 2:
        Code #: 3001
        Filename:  /Users/xyz/sample/te.js
        Caption: Potentially Weak Password
        Category: password-secret
        Line #: 1
        Value: password = "Prvdfgfsgd"
        Severity: high
        Confidence: medium
        Labels: weak password
        Associated CWEs: CWE-798/CWE-312/CWE-257/CWE-259

        ***** Total issues found *****
            2 Potentially Weak Password
            2 TOTAL ISSUES
2023/02/14 16:23:19 
1 files scanned in 1.867156ms
2023/02/14 16:23:19 
33 rules observed
Scan detected findings above the accepted threshold -- Failing.
exit status 1

Ignore false positive string

I see that we have a way of ignoring a file. Can we introduce a way to ignore a string as well?

example: in my .env.example I have placeholders

my_secret=ThisIsASecretToReplace

I want them to see this in the first run and then add "ThisIsASecretToReplace" to an exception list. By doing this, it still forces them to think about the data they are putting in .env.example and will always require the initial review of the finding. Currently I have to ignore the file .env.example altogether which means if someone actually puts a secret in there that is valid then no one will be monitoring (outside of the PR review).

Reporting incorrect finding for SSN

As per https://www.codeproject.com/Articles/651609/Validating-Social-Security-Numbers-through-Regular

A Social Security number CANNOT:

Contain all zeroes in any specific group (ie 000-##-####, ###-00-####, or ###-##-0000)
Begin with ’666′.
Begin with any value from ’900-999′
Be ’078-05-1120′ (due to the Woolworth’s Wallet Fiasco)
Be ’219-09-9999′ (appeared in an advertisement for the Social Security Administration)

But today, it's reporting SSN 9XX-XX-XXXX as valid finding.

Lack of GitHub releases / precompiled binaries

GitHub releases would allow users to download prebuilt versions of the application without having to compile it. This is especially useful for automated CI pipelines, where having to setup a Go environment and compile EarlyBird can add unnecessary overhead.

Please publish compiled binaries of EarlyBird via GitHub releases to allow them to be downloaded as part of CI pipelines. This could be implemented in a CI/CD pipeline of your choice - I'll open a PR to do it in GitHub Actions since that's probably the easiest and doesn't require any external services.

Incorrect path to file in case of keys detection

Problem

Files containing keys are not reported with full path to file, only file name is printed out in JSON report (haven't tested other types of report).

Example

"caption": "Key database file",
"filename": "keystore.jks",

instead of:

"caption": "Key database file",
"filename": "src/resources/keystore.jks",

Additional information

Files containing other type of issues point directly to the right place:

"caption": "Potential password in file",
"filename": "war/src/broken/resources/runtime.properties",

Contradictory not all key files are reported that way:

"caption": "Potential repository key in file",
"filename": "frontend/properties.toml",

Problem applies to rules like:

"caption": "Keychain database file",
"caption": "Potential cryptographic key bundle",
"caption": "Key database file",

and similar.

Base directory should be ignored during ignore matching

This is rather an enhancement than a bug, but makes things more clear

As of now whole path is being checked against ignorefile
BUT
Imagine the situation that:

Ignore file contains entry "/test/"
User uses his /var/lib/test/projects/ directory to download his projects into
Then when Earlybird is executed with parameter: -path " /var/lib/test/projects/" it will ignore all project files, nothing will be scanned

Proposed remediation:
Base directory path should be removed from matching against ignored patterns

meta: Comparison to gitleaks?

👋 Cool project! I was curious how earlybird compares to other projects like gitleaks? I see earlybird can scan a bit more types of targets, but are the patterns both recognize the same? I've used gitleaks for a while and am curious to adopt both tools.

Project: https://github.com/zricethezav/gitleaks

Shell scripts are not executable

Both build.sh and install.sh are not executable as the #! is required to be the first line in the file, however it has been displaced by the licence string.

Attempting to execute them results in the following error (on macOS):

Failed to execute process './build.sh'. Reason:
exec: Exec format error
The file './build.sh' is marked as an executable but could not be run by the operating system.

Cloning of git repository leaves files in temporary directory

After data is being downloaded using -git flag to temporary directory it is not being removed after data check.
I think that the expected behavior should be that the data is removed from the temporary location after checking presence of sensitive data.

The problem is that if I am running check against many code repositories temp directory grows significantly and I need to have additional monitoring activity to erase temporary data.

Version 1.24.6

Ignorefile case sensitivity is broken

.ge_ignore file case sensitivity works in a weird way. At least on Windows machine. Entries in ignore file have to be put in lower case to match something.

The problem is that when in ignore file is entry: '*.txt' it matches any *.txt files (i.e. Readme.txt, readme.TXT) which is good behavior.
But when I put entry "*.TXT" it does not match anything (at least not expecting readme.TXT).
It showed when I tried to exclude some path which contained word "Libs" (i.e. "Libs/sweet.lib"). When I tried with all combinations of "*/Libs/*", "*Libs*", "Libs*" nothing worked. Only "*/libs/*" was matching.

This is not a huge problem, but it is a very counter intuitive behavior.

Version: 1.24.6

if fail-severity flag is optional return exit 0

The default value for --fail-severity is set to low and even when it is not set earlybird seems to return exit 1. There should be a way to say display particular severity but donot return exit 1 as i haven't defined fail-severity.

Create 2 new custom allowable values

Our API product documentation needs two lowercase alphanumeric allowable values for two specific data fields: access_token and refresh_token in an 8-4-4-4-12 format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.

EXAMPLE: 50742b22-cfe6-495d-98cc-7fb1be768a3d

Build fails on macOS

When running build.sh, the build fails with the following log:

Running Unit Tests
go: downloading golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e
go: downloading github.com/gorilla/mux v1.7.4
go: downloading github.com/gocarina/gocsv v0.0.0-20200330101823-46266ca37bd3
go: downloading gopkg.in/src-d/go-git.v4 v4.13.1
go: downloading github.com/dghubble/sling v1.3.0
go: downloading github.com/howeyc/gopass v0.0.0-20190910152052-7cb4b85ec19c
go: downloading github.com/google/go-github v17.0.0+incompatible
go: downloading golang.org/x/text v0.3.2
go: downloading golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4
go: downloading github.com/google/go-querystring v1.0.0
go: downloading github.com/sergi/go-diff v1.0.0
go: downloading gopkg.in/src-d/go-billy.v4 v4.3.2
go: downloading github.com/kevinburke/ssh_config v0.0.0-20190725054713-01f96b0aa0cd
go: downloading github.com/xanzy/ssh-agent v0.2.1
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/emirpasic/gods v1.12.0
go: downloading golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd
go: downloading github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99
go: downloading github.com/src-d/gcfg v1.4.0
go: downloading gopkg.in/warnings.v0 v0.1.2
# github.com/americanexpress/earlybird/pkg/api
pkg/api/api.go:72:19: conversion from Duration (int64) to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
pkg/api/api.go:141:19: conversion from Duration (int64) to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
# github.com/americanexpress/earlybird/pkg/core
pkg/core/core.go:261:19: conversion from Duration (int64) to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
# github.com/americanexpress/earlybird/pkg/writers
pkg/writers/jsonout_test.go:55:17: conversion from Duration (int64) to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
Unit Tests FAILED!
FAIL	github.com/americanexpress/earlybird/pkg/api [build failed]
ok  	github.com/americanexpress/earlybird/pkg/config	0.325s
FAIL	github.com/americanexpress/earlybird/pkg/core [build failed]
Failed to open ignore file open .ge_ignore: no such file or directory
Failed to open ignore file open /Users/phil/.ge_ignore: no such file or directory
Failed to open ignore file open .ge_ignore: no such file or directory
--- FAIL: Test_isIgnoredFile (0.00s)
    --- FAIL: Test_isIgnoredFile/Check_if_file_is_ignored (0.00s)
        fileUtil_test.go:148: isIgnoredFile() = false, want true
FAIL
FAIL	github.com/americanexpress/earlybird/pkg/file	0.222s
ok  	github.com/americanexpress/earlybird/pkg/git	0.487s
ok  	github.com/americanexpress/earlybird/pkg/postprocess	0.209s
ok  	github.com/americanexpress/earlybird/pkg/scan	0.254s
ok  	github.com/americanexpress/earlybird/pkg/update	0.468s
ok  	github.com/americanexpress/earlybird/pkg/utils	0.252s
ok  	github.com/americanexpress/earlybird/pkg/wildcard	0.184s
FAIL	github.com/americanexpress/earlybird/pkg/writers [build failed]
FAIL

This occurs on macOS version 10.15.5, and Go version go1.15.2 darwin/amd64.

Wrong verbose information about scanned files

When I scan folder with 4 files I see in log:

Reading file
Reading file /mnt/c/git-repo/earlybird/verify/.ge_ignore
Reading file /mnt/c/git-repo/earlybird/verify/checkfile.properties
Reading file /mnt/c/git-repo/earlybird/verify/checkfile2.properties
***** Total issues found *****
0 TOTAL ISSUES

4 files scanned in 18.7711ms

The first entry is "Reading file" without the data of the file in folder.

Feature Request: GitHub Action for EarlyBird

Couldn't find a Discord/Slack/etc to post this to so I will post this here. It would be amazing if this repo could support a GitHub Action so that we can bake this into our CI/CD easily.

ignore files issue

I'm testing earlybird against an ansible role repository, normally an easy one.
I still get many false-positive matches that I'm trying to ignore zith ~/.ge_ignore but it seems the pattern style is not working or a format that I'm not expecting.
Tried with filename, shell wildcard pattern. also double wildcard like https://github.com/aschaef19/earlybird/blob/main/.ge_ignore

$ go-earlybird -path=. -verbose
022/04/30 10:08:10 Go-EarlyBird version:  2.0.0
Severity Fail threshold (at or above):  low
Confidence Fail threshold (at or above):  low
Severity Display threshold (at or above):  low
Confidence Display threshold (at or above):  low
Max file size to scan:  10240000  bytes
2022/04/30 10:08:10 loading module:  ccnumber
2022/04/30 10:08:10 loading module:  content
2022/04/30 10:08:10 loading module:  filename
2022/04/30 10:08:10 loading module:  inclusivity-rules
2022/04/30 10:08:10 loading module:  password-secret
2022/04/30 10:08:10 Scanning directory:  .
2022/04/30 10:08:10 Ignore pattern:  *.git/*, .vagrant/, *.retry, .kitchen, inspec.lock, [._]*.s[a-v][a-z], [._]*.sw[a-p], [._]s[a-v][a-z], [._]sw[a-p], Session.vim, Sessionx.vim, .netrwhist, *~, tags, [._]*.un~, __pycache__/, *.py[cod], *$py.class, *.so, .Python, build/, develop-eggs/, dist/, downloads/, eggs/, .eggs/, lib/, lib64/, parts/, sdist/, var/, wheels/, *.egg-info/, .installed.cfg, *.egg, MANIFEST, *.manifest, *.spec, pip-log.txt, pip-delete-this-directory.txt, htmlcov/, .tox/, .coverage, .coverage.*, .cache, nosetests.xml, coverage.xml, *.cover, .hypothesis/, *.mo, *.pot, *.log, .static_storage/, .media/, local_settings.py, instance/, .webassets-cache, .scrapy, docs/_build/, target/, .ipynb_checkpoints, .python-version, celerybeat-schedule, *.sage.py, .env, .venv, env/, venv/, ENV/, env.bak/, venv.bak/, .spyderproject, .spyproject, .ropeproject, /site, .mypy_cache/, .DS_Store, .AppleDouble, .LSOverride, Icon, ._*, .DocumentRevisions-V100, .fseventsd, .Spotlight-V100, .TemporaryItems, .Trashes, .VolumeIcon.icns, .com.apple.timemachine.donotpresent, .AppleDB, .AppleDesktop, Network Trash Folder, Temporary Items, .apdisk, *~, .fuse_hidden*, .directory, .Trash-*, .nfs*, Thumbs.db, ehthumbs.db, ehthumbs_vista.db, *.stackdump, [Dd]esktop.ini, $RECYCLE.BIN/, *.cab, *.msi, *.msm, *.msp, *.lnk, secring.*, *.ca, *.crt, *.csr, *.der, *.kdb, *.org, *.p12, *.pem, *.rnd, *.ssleay, *.smime, **/.git, **/.gitignore, **/.github/workflows/galaxy.yml, **/.secrets.baseline, **/.pre-commit-config.yaml, **/test/earlybird/falsepositives-ansible.yaml, .git, .gitignore, .github/workflows/galaxy.yml, galaxy.yml, .secrets.baseline, .pre-commit-config.yaml, test/earlybird/falsepositives-ansible.yaml, falsepositives-ansible.yaml, */.git, */.gitignore, /.git, /.gitignore, */.git*, */.gitignore*, ./.git, ./.gitignore
2022/04/30 10:08:10 Reading file  .ansible-lint
2022/04/30 10:08:10 Reading file  .codespellignore
2022/04/30 10:08:10 Reading file  .git
2022/04/30 10:08:10 Reading file  .github/stale.yml
2022/04/30 10:08:10 Reading file  .github/workflows/codespell.yml
2022/04/30 10:08:10 Reading file  .github/workflows/default.yml
2022/04/30 10:08:10 Reading file  .github/workflows/dryrun-bare.yml
2022/04/30 10:08:10 Reading file  .github/workflows/earlybird.yml
2022/04/30 10:08:10 Reading file  .github/workflows/galaxy.yml
2022/04/30 10:08:10 Reading file  .github/workflows/lint.yml
2022/04/30 10:08:10 Reading file  .gitignore
[...]

from my reading of code, "Reading file " should not appear if file is correctly ignored
https://github.com/americanexpress/earlybird/blob/main/pkg/file/fileUtil.go#L196
match seems custom character per character as per
https://github.com/americanexpress/earlybird/blob/main/pkg/wildcard/patternMatch.go

Note that even if it says " Go-EarlyBird version: 2.0.0", this is from latest download aka https://github.com/americanexpress/earlybird/releases/download/v3.12.0/go-earlybird-linux

Example run in https://github.com/juju4/ansible-adduser/runs/6142207431?check_suite_focus=true#step:6:1

Thanks for sharing your work

Error: invalid memory address or nil pointer dereference

I have just built the binaries from the source code (both linux/amd64 and windows/amd64 behave the same way).
When executing it I get as a result:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xa12032]

goroutine 1 [running]:
github.com/americanexpress/earlybird/pkg/core.(*EarlybirdCfg).GetRuleModulesMap.func1(0xc000026f60, 0x19, 0x0, 0x0, 0xc47f00, 0xc00010ed80, 0xc000107be8, 0x4108ad)
/var/lib/jenkins/workspace/Earlybird-build-binaries/earlybird/pkg/core/core.go:148 +0x32
path/filepath.Walk(0xc000026f60, 0x19, 0xc000107c30, 0xc000026f60, 0x19)
/usr/lib/golang/src/path/filepath/path.go:404 +0x6b
github.com/americanexpress/earlybird/pkg/core.(*EarlybirdCfg).GetRuleModulesMap(0x1044e80, 0x10445e0, 0xc00002c810)
/var/lib/jenkins/workspace/Earlybird-build-binaries/earlybird/pkg/core/core.go:147 +0xef
github.com/americanexpress/earlybird/pkg/core.(*EarlybirdCfg).ConfigInit(0x1044e80)
/var/lib/jenkins/workspace/Earlybird-build-binaries/earlybird/pkg/core/core.go:176 +0x2ef
main.main()
/var/lib/jenkins/workspace/Earlybird-build-binaries/earlybird/go-earlybird.go:47 +0x35d

Need more details on HTTP API usage

Is docker image available for this?
Can you please provide details on how to use HTTP APIs with sample payloads?

Name scanner is failing scan if severity is adjusted to low/medium for a file name.

If we pass --fail-severity=high and --fail-confidence=high config, then results are not what we are expecting for file name scanner.

Actual result

Severity Fail threshold (at or above):  high
Confidence Fail threshold (at or above):  high
Severity Display threshold (at or above):  high
Confidence Display threshold (at or above):  high
Max file size to scan:  10240000  bytes
2023/09/15 17:34:43 Go-EarlyBird version:  dev
2023/09/15 17:34:43 loading module:  content
2023/09/15 17:34:43 loading module:  filename
2023/09/15 17:34:43 loading module:  inclusivity-rules
2023/09/15 17:34:43 loading module:  password-secret
2023/09/15 17:34:43 loading module:  amex-common
2023/09/15 17:34:43 loading module:  amex-miscellaneous
2023/09/15 17:34:43 loading module:  amex-password-secret
2023/09/15 17:34:43 loading module:  ccnumber
2023/09/15 17:34:43 Scanning directory:  /tmp/data
2023/09/15 17:34:43 Failed to open ignore file open /home/golang/.ge_ignore: no such file or directory
2023/09/15 17:34:44 
368 files scanned in 1.3656482s
2023/09/15 17:34:44 
87 rules observed
Scan detected findings above the accepted threshold -- Failing.
	***** Total issues found *****
	    0 TOTAL ISSUES

Expected result

Severity Fail threshold (at or above):  high
Confidence Fail threshold (at or above):  high
Severity Display threshold (at or above):  high
Confidence Display threshold (at or above):  high
Max file size to scan:  10240000  bytes
2023/09/15 17:34:43 Go-EarlyBird version:  dev
2023/09/15 17:34:43 loading module:  content
2023/09/15 17:34:43 loading module:  filename
2023/09/15 17:34:43 loading module:  inclusivity-rules
2023/09/15 17:34:43 loading module:  password-secret
2023/09/15 17:34:43 loading module:  amex-common
2023/09/15 17:34:43 loading module:  amex-miscellaneous
2023/09/15 17:34:43 loading module:  amex-password-secret
2023/09/15 17:34:43 loading module:  ccnumber
2023/09/15 17:34:43 Scanning directory:  /tmp/data
2023/09/15 17:34:43 Failed to open ignore file open /home/golang/.ge_ignore: no such file or directory
2023/09/15 17:34:44 
368 files scanned in 1.3656482s
2023/09/15 17:34:44 
87 rules observed
	***** Total issues found *****
	    0 TOTAL ISSUES

Embedding default configurations in released binary

Hey there 👋 Love the project and thanks for all your work here!

Suggestion

For all files in the config/ directory, I think this is a perfect use case to embed your configurations in the binary utilizing go:embed.

This would essentially alleviate the requirement for users to install the repository locally to setup the go-earlybird project using your shell scripts because the binary would already have the configs packaged in via embed.FS rather than reading the user's local filesystem for the baseline configs. This then opens up the door for you to distribute an easily packagable go get command, brew install or whatever package manager user's would like because everything they need is right there in the executable.

This could be the default and then allow users to provide additional configs if they choose to by reading the ~/.go-earlybird/foo directory. As mentioned here in your utils.go

Specifics

Referenced Earlybird Config Directory
Additional documentation on embed.FS

So right here where you are reading in the config with an os.Open, if you switch this to use the new embed.FS, you'll get this end result.

Current

//LoadConfig parses json configuration file into structure
func LoadConfig(cfg interface{}, path string) (err error) {

	jsonFile, err := os.Open(path)
	// if we os.Open returns an error then handle it
	if err != nil {
		return err
	}
...

Proposed Change

+ //go:embed config/
+var embeddedConfig embed.FS

//LoadConfig parses json configuration file into structure
func LoadConfig(cfg interface{}, path string) (err error) {

+	jsonFile, err := embeddedConfig.Open(path)
	// if we embeddedConfig.Open returns an error then handle it
	if err != nil {
		return err
	}
...

outdated replace usage in go.mod

There are replace lines in go.mod pinning golang.org/x/text to v0.3.3. However, according to GO-2022-1059, version of golang.org/x/text before v0.3.8 suffers potential vulnerability. Should them be updated?

https://github.com/americanexpress/earlybird/blob/main/go.mod

replace (
	golang.org/x/text v0.3.0 => golang.org/x/text v0.3.3
	golang.org/x/text v0.3.2 => golang.org/x/text v0.3.3
)

	func GetBBProject(bbURL string) (project string) {
	re := regexp.MustCompile(`(?:projects/)([^/]*)`)