thoughtworks / talisman Goto Github PK

Hi, first of all thx for this great tool!

I noticed that talisman fails to detect passwords like this:

$ pwgen -1 -ys 36 | tee test-pw
Jz&LBE{b7J.SKn;%w1yJ-gqHQoHe_~g7l+:<
$ git add test-pw
$ git commit -m'test'

 …succeeds

I have no clue of talimans algoryhtm to detect secret material, but this case shold be covers (a file only consisting of high entropy random characters).

Like .talismanignore is used for ignoring the files. Add new filename for detection using new file .talismanadd

Homebrew based installation would make it much simpler to install talisman on MacOS.

While running the uninstall script, it is not able to delete the hooks created by talisman.

for example, to explicitly install to a repo or as a git hook template

It would be more inviting if a description would be set, and give people an idea of what they are looking at.

When I create a new branch locally and want to push it, setting the upstream fails:

$ git push --set-upstream origin my-new-branch

(the operation just hangs)

Steps to reproduce

$ git checkout -b my-new-branch
$ echo "foo" > file.txt
$ git add file.txt
$ git commit -m "Test"
$ git push --set-upstream origin my-new-branch

Context

System: MacOS 10.13.2
Shell: zsh 5.3
Git: 2.16.0

When I follow the instructions in the docs, I get the following error:
╰─➤ godep restore
godep: Dep (github.com/Sirupsen/logrus) restored, but was unable to load it with error:
Package (golang.org/x/sys/unix) not found
godep: Error checking some deps.

Work-around was to first:
go get golang.org/x/sys/unix

I'm not sure what the right way to solve that really is, so I'll defer to your recommendation rather than make PR.

I had half forgotten that I had installed Talisman, then I tried pushing a file containing a lot of hashes (sadly, d3.js is not Talisman's friend) and up popped a pile of warnings with no indication of what was causing them. For a second I thought it might be something on the remote github server, until I remembered that I had Talisman installed globally.

It would be good if Talisman reported that it was the source of the warnings, possibly with a link to the github repo.

because ...

• easier on-boarding of talisman on to distribution platforms like HomeBrew
• generally a good idea to know which build of talisman is installed or in use

This is a file that we need to check in, but it is disallowed. Right now I just temporarily remove talisman in order to check in schema changes.

The following errors were detected in src/main/resources/schema.sql
     The file name "src/main/resources/schema.sql" failed checks against the pattern \bsql|sql\b

Cade:

The paragraph on the landing page https://thoughtworks.github.io/talisman/ could be reduced if you want to get people's attention. You could probably make this a single sentence (see next comment) followed by your "Learn more link". That way the viewer sees a brief statement and a quick example of it in action.

The statement "is a tool *is *to validate code changes" doesn't tell your reader what it does. What kind of validation? It's validating code changes to make sure secrets aren't accidentally checked in. (also, bold emphasis on an extra word)

Bitbucket server has the possibility to apply commit hooks (pre/post) as the repository and the project level(affecting all repositories).
It would be great if talisman would be available as a plugin so you will not have to be applied to every repository individually.
Applying plugins at the project level allows you to configure a plugin at a bigger scope and then disable or configure it differently for a repository.

Hi guys,
Any plans on releasing version for windows?

I upgraded to the latest Talisman. My project has files that contains hex encoded text. Specifically, the project.pbxproj inside the xcodeproj directory for an iOS code base.

I added it to .talismanrc as suggested. I added .talismanrc to git and tried to commit it. Now I get an error saying that .talismanrc has a hex encoded file. If I add it to the .talismanrc file, then it would change it's checksum and it won't work.

The solution is to NOT git add .talismanrc and add it to .gitignore. However, the error message asks you to add .talismanrc to itself.

The error message should ask you to NOT add .talismanrc to git and add it to .gitiignore instead.

Hello everyone,

After running go lint for git_repo package gives following lint issues:

git_repo/git_repo.go:1:1: don't use an underscore in package name
git_repo/git_repo.go:40:1: exported method GitRepo.StagedAdditions should have comment or be unexported
git_repo/git_repo.go:59:1: comment on exported method GitRepo.AdditionsWithinRange should be of the form "AdditionsWithinRange ..."
git_repo/git_repo.go:108:9: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
git_repo/git_repo_internal_test.go:1:1: don't use an underscore in package name
git_repo/git_repo_test.go:1:1: don't use an underscore in package name

We should adhere to go code style guidelines. I would love to raise PR after having discussion.

Currently, Talisman only scans the file which are being modified as part of a commit. This change is to ensure that Talisman is capable of scanning all the files at the repo level.

Currently, pre-commit tool can be used for supporting hook chaining in *nix machines.
However, it does not work with Windows.
It might be useful to support a tool such as Husky for this purpose.

I am getting this error when I am trying to run ~/install-talisman.sh

When filenames are added to talismanignore, they currently are ignored for good.
This runs a risk since the files could introduce secrets later, which would still get ignored.
A checksum on the files mentioned in the talismanignore could monitor this. If contents change don’t ignore. That could be useful for files - every time someone changes it, they have to double check that they are not checking in sensitive info

I've tried to push 3 commits, which failed with the following error.

There's nothing really irregular in particular which is causing this. My changeset included a package-lock.json file with others that are called out in .talismanignore.

goroutine 1 [running]:
runtime.Gosched()
    /opt/golang/src/runtime/proc.go:166 +0x14 fp=0xc820043638 sp=0xc820043628
runtime.gcAssistAlloc(0x10000, 0xc81ffdab01)
    /opt/golang/src/runtime/mgcmark.go:277 +0x204 fp=0xc820043698 sp=0xc820043638
runtime.mallocgc(0x10000, 0x1422e0, 0x1, 0x12c31)
    /opt/golang/src/runtime/malloc.go:711 +0x535 fp=0xc820043768 sp=0xc820043698
runtime.newarray(0x1422e0, 0xff82, 0x12d0b)
    /opt/golang/src/runtime/malloc.go:777 +0xc9 fp=0xc8200437a8 sp=0xc820043768
runtime.makeslice(0x13a300, 0x0, 0xff82, 0x0, 0x0, 0x0)
    /opt/golang/src/runtime/slice.go:32 +0x165 fp=0xc8200437f8 sp=0xc8200437a8
io/ioutil.readAll(0x4b4388, 0xc8204a2010, 0xff82, 0x0, 0x0, 0x0, 0x0, 0x0)
    /opt/golang/src/io/ioutil/ioutil.go:19 +0x9d fp=0xc8200438f0 sp=0xc8200437f8
io/ioutil.ReadFile(0xc82049e200, 0x3d, 0x0, 0x0, 0x0, 0x0, 0x0)
    /opt/golang/src/io/ioutil/ioutil.go:70 +0x19f fp=0xc8200439b8 sp=0xc8200438f0
github.com/thoughtworks/talisman/git_repo.GitRepo.ReadRepoFile(0xc82000a0c4, 0x1f, 0xc8200a30bc, 0x1d, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/jack/gohome/src/github.com/thoughtworks/talisman/git_repo/git_repo.go:85 +0x115 fp=0xc820043a40 sp=0xc8200439b8
github.com/thoughtworks/talisman/git_repo.GitRepo.AdditionsWithinRange(0xc82000a0c4, 0x1f, 0xc8200920cd, 0x28, 0xc820092092, 0x28, 0x0, 0x0, 0x0)
    /home/jack/gohome/src/github.com/thoughtworks/talisman/git_repo/git_repo.go:63 +0x1ac fp=0xc820043c20 sp=0xc820043a40
main.(*PrePushHook).getRepoAdditions(0xc820043ea0, 0x0, 0x0, 0x0)
    /home/jack/gohome/src/github.com/thoughtworks/talisman/pre_push_hook.go:70 +0xaf fp=0xc820043c90 sp=0xc820043c20
main.(*PrePushHook).GetRepoAdditions(0xc820043ea0, 0x0, 0x0, 0x0)
    /home/jack/gohome/src/github.com/thoughtworks/talisman/pre_push_hook.go:56 +0xc39 fp=0xc820043d38 sp=0xc820043c90
main.run(0x4b4388, 0xc820030008, 0x2c9200, 0x1c2970, 0x8, 0x1e1e80)
    /home/jack/gohome/src/github.com/thoughtworks/talisman/talisman.go:67 +0x4c7 fp=0xc820043ee8 sp=0xc820043d38
main.main()
    /home/jack/gohome/src/github.com/thoughtworks/talisman/talisman.go:45 +0x158 fp=0xc820043f50 sp=0xc820043ee8
runtime.main()
    /opt/golang/src/runtime/proc.go:111 +0x2b0 fp=0xc820043fa0 sp=0xc820043f50
runtime.goexit()
    /opt/golang/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820043fa8 sp=0xc820043fa0
error: failed to push some refs to 'ssh://[email protected]_domain:1234/dir/my-project.git'

for example, if PASSWORD is present in a file?

Users should have the ability to ignore specific lines within files which match a certain sensitive content check (format should be: <file_name> <regex_to_ignore>)

S3 is having issues today. When I ran the install script it sat on "Downloading and verifying" for a couple minutes and then exited silently. I didn't realize until I dug around that it was related to the outage.

Hey folks,
I want to add talisman as a stand alone scanner (not on a hook), so that I can run it as part of a tool I'm building (https://github.com/Stono/hawkeye).

Is this possible?

Thanks for talisman! To make it even easier to install it across all repositories, I'd like an option to install talisman for existing Git clones (not just future ones).

Whether that is just via extending documentation, with e.g. the following snippet, or as a automatic step when running the install script outside of a Git repo, surely both would be an improvement.

$ find ~/my_projects -name ".git" -exec sh -c 'pushd {}/.. && ~/install-talisman.sh && popd' \;

On our project, we are using git-crypt to safely store privileged material. Talisman is giving off false positives in that case, rendering it almost useless (as an exclusion list has to be kept and maintained manually, which could make things even less safe due to confusion).

Talisman is a great basic tool, but it doesn't play nicely with other frameworks, and as mentioned in #35 and #26 there is interest in using it standalone, or with other tools.

The pre-commit framework (https://github.com/pre-commit/pre-commit) is an open-source framework for defining and combining multiple pre-commit hooks for a project. It would be extremely convenient to be able to use talisman within this framework - and it could also simplify talisman by allowing pre-commit to take care of the git integration side of things.

Talisman is great, however the documentation should get improved.
I don't understand the difference between a .talismanrc and a .talismanignore file.

Also:

• What's the config file formats ?
• Please add example files with all possible options together with proper explanations
• How to use patterns in these config files
• How to ignore complete directories

.talismanrc could replace or augment/accompany .talismanignore (One file seems better, but ... SRP)
Talisman could be used to create and modify the file.
The file could be yaml or json.

We can club issues #47 by @AikoPath, #39 by @alesti, #33 by @abhsk and with this issue.

Please see my discussion with @flosell on commits for #38 for some more context/ideas.

When I try to curl the install script, as is shown on talisman's webpage, I get an 404 error

https://thoughtworks.github.io/talisman/install.sh

Two questions:

1. How do I remove talisman inside my project, i.e. locally?
2. How do I remove talisman globally?

I succeeded to push a DSA key named id_dsa_wgarcia in my repo, a possible fix is to replace this regexp ^.+_dsa$ by this one ^.+_dsa.*$ in https://github.com/thoughtworks/talisman/blob/master/detector/filename_detector.go#L20

Hi,
I have a lot of LaTeX documentation in a git repo which refers e.g. to gdrive docs or sheets.
Talisman does a good job to identify this referrers, because the looking like base64 code :-)

I am able to disarm only files with this patterns.
I'd like to disarm only the pattern itself, not the hole file - so i will stopped, if i do another mistake in the same tex file.

Same problem may occur in comments of any programming language.

Aleks

Gopkg.lock file track and locks the dependencies and talisman wrongly detects the error in this file and doesn't allow to push the commit

The following errors were detected in Gopkg.lock
         Expected file to not to contain base64 or hex encoded texts such as: "1:ef5b0622d834c139454148b8fd0c92bb314828900532b267ae62da9fec109866"
         Expected file to not to contain base64 or hex encoded texts such as: "f0300d1749da6fa982027e449ec0c7a145510c3c"

The Readme talks about downloading the Talisman binary. I can't see where it is?

touch secret.pem
git commit -am "whoops"
git push
talisman: booo
rm secret.pem
git commit -am "all safe now"
git push
talisman: 👍

Talisman blocked default README.md from "create-react-app" but allowed to push when I cleared README.md in second commit.

Example here:
https://github.com/mariusz-zawadzki/montessori-teacher-tools/commits/master

Steps to recreate:

1. Init a repo
2. add a file that has content that is not allowed.
3. git commit
4. git push (get failure)
5. Clear/fix content of said file.
6. git commit
7. git push (success, expected failure)

Recently the shared space I work out of has been hitting the Github api request limit. (This happens more frequently in corporations where many share a single external IP) When this happens the script exits with no more information.

I was able to get around this by adding my : manually to curl commands in the installation script. Maybe we want to add support for an env variable that will allow people to authenticate?

Talisman fails to detect this yaml file:

room: [email protected]
username: [email protected]
password: eytgdtewqy

Right now, it looks like talisman can only be used as a git hook (pre-push or pre-commit). It might be good to split out the actual detector functionality and make it more reusable. I'm thinking of things like running it on demand on a repo, a set of commits, on CI, and so on. What do you think?

When another change has been made on the remote I'm pushing to (requiring a git pull) talisman fails in quite an ugly way:

FATA[0000] Command execution failed                      error=exit status 128 output=
error: failed to push some refs to '[email protected]:jacksingleton/push-test.git'

When I have a false positive on an environment config file I can only .talismanignore the complete file.
At this point, talisman has mostly become useless for the given repository. Since the one file that has the highest probability where I accidentally have credentials stored would be that environment config file.

This has happened multiple times to me. I don't remember the exact use cases but something like google analytics tracking id or something else that looks like a secret that I need to send to my Frontend.

If I could ignore the exact string value of the false positive I could continue to get the file checked for future pushes while also push when the false positive comes up.

This is related to / another reason for #39.