sensepost / gowitness Goto Github PK

I don't see a way to use this on a website that has a login page. Is there a way to provide login credentials to this utility?

Screenshot failed !!!!

ERRO[2020-09-06 15:26:48] An error occurred while trying to get the Chrome version chrome-path=/usr/bin/chromium-browser err="exit status 1"

Describe the bug
Trying to screen shot single url

To Reproduce
Steps to reproduce the behavior:
1.Type this in terminel and get the following error
gowitness single --url=https://www.facebook.com/
Error
gowitness single --url=https://www.facebook.com/
ERRO[2020-04-01 15:32:28] Failed to query url error="[context deadline exceeded (Client.Timeout or context cancellation while reading body)]" url="https://www.facebook.com/"
INFO[2020-04-01 15:32:28] Complete run-time=3.1479092s
Screenshots
If applicable, add screenshots to help explain your problem.

Version Information:
Ubuntu WSL (Windows)

Great tool!

Just a word about the license, though, Creative Commons isn't really meant to be used for software. They even explicitly recommend against it:

https://creativecommons.org/faq/#can-i-apply-a-creative-commons-license-to-software

I'd love to make some changes / features / etc..., but things like the non-commercial clause of the CC license here make that problematic.

Thanks!

Hi,

Can't thank you enough for come up with this tool!

A last release, there's a but where when you specify screenshots folders and generate a report images does not reflect at db. As a result, screenshots won't load.

I'm using --source flag with urls and -d ./screenshots/ flag.

Cheers

First off, great tool. Is there a way to scan multiple CIDR ranges at once? I tried separating by commas and spaces but seems to be a no go. I guess I can just write up a quick for loop... Thanks again.

Describe the bug
I am experiencing two errors when trying to run gowitness. One error says something about a "context deadline exceeded", and the other is "failed to save screenshot buffer"

To Reproduce
Steps to reproduce the behavior:
On the command line, type:

1. gowitness single https://tesla.com
2. gowitness single https://reddit.com

Expected behavior
Screenshots will be taken of the site.

Screenshots

Version Information:

• OS: Kali GNU/Linux Rolling x86_64
• gowitness: 2.0.0

Hi guys,

Hope u are all well !

I tried to get a screenshot without the cli.

package main

import (
	//"fmt"
	//"os"
	"net/url"
	"strconv"
	"strings"
	"time"

	//homedir "github.com/mitchellh/go-homedir"
	log "github.com/sirupsen/logrus"
	//"github.com/spf13/cobra"
	//"github.com/spf13/viper"

	chrm "github.com/sensepost/gowitness/chrome"
	"github.com/sensepost/gowitness/storage"
	"github.com/sensepost/gowitness/utils"
)

func main() {

	// Init Google Chrome
	chrome = chrm.Chrome{
		Resolution:    resolution,
		ChromeTimeout: chromeTimeout,
		Path:          chromePath,
		UserAgent:     userAgent,
	}
	chrome.Setup()

	screenshotURL := "https://www.zataz.com/osint-des-numeros-de-politiques-francaises-interceptes-via-whatsapp/"
	u, err := url.ParseRequestURI(screenshotURL)
	if err != nil {
		log.WithField("url", screenshotURL).Fatal("Invalid URL specified")
	}

	// Process this URL
	utils.ProcessURL(u, &chrome, &db, 3000)

	log.WithFields(log.Fields{"run-time": time.Since(startTime)}).Info("Complete")
}

var (
	cfgFile    string
	chrome     chrm.Chrome
	db         storage.Storage
	dbLocation string

	// logging
	logLevel  string
	logFormat string

	// 'global' flags
	waitTimeout   int
	resolution    string
	chromeTimeout int
	chromePath    string
	userAgent     string

	// screenshot command flags
	screenshotURL         string
	screenshotDestination string

	// file scanner command flags
	sourceFile string
	maxThreads int

	// range scanner command flags
	scanCidr           []string
	scanFileCidr       string
	scanPorts          string
	skipHTTP           bool
	skipHTTPS          bool
	randomPermutations bool

	// generate command
	reportFileName string

	// execution time
	startTime = time.Now()

	// version
	version = "1.0.8"
)

// initLogging prepares the logrus logger and format.
// the --log-level and --log-format commandline args lets you
// control what and how logrus outputs data.
func initLogging() {

	switch logLevel {

	case "debug":
		log.SetLevel(log.DebugLevel)

	case "info":
		log.SetLevel(log.InfoLevel)

	case "warn":
		log.SetLevel(log.WarnLevel)

	case "error":
		log.SetLevel(log.ErrorLevel)

	case "fatal":
		log.SetLevel(log.FatalLevel)

	default:
		log.WithField("log-level", logLevel).Warning("invalid log level. defaulting to info.")
		log.SetLevel(log.InfoLevel)
	}

	// Include timestamps in the text format output
	textformat := &log.TextFormatter{
		FullTimestamp:   true,
		TimestampFormat: "2006-01-02 15:04:05",
	}

	switch logFormat {

	case "text":
		log.SetFormatter(textformat)

	case "json":
		log.SetFormatter(new(log.JSONFormatter))

	default:
		log.WithField("log-format", logFormat).Warning("invalid log format. defaulting to text.")
		log.SetFormatter(textformat)
	}
}

// Checks if some of the globally provided arguments are valid.
func validateFlags() {

	// Check screenresolution argument values
	parsedResolution := strings.Split(resolution, ",")
	if len(parsedResolution) != 2 {

		log.WithField("resolution", resolution).Fatal("Invalid resolution value provided")
	}

	if _, err := strconv.Atoi(parsedResolution[0]); err != nil {
		log.WithField("resolution", resolution).Fatal("Failed to parse resolution x value")
	}

	if _, err := strconv.Atoi(parsedResolution[1]); err != nil {
		log.WithField("resolution", resolution).Fatal("Failed to parse resolution y value")
	}

}

What do I miss ?

Cheers,
L.

It looks like gowitness on macOS 10.12.6 (16G1212) thinks my installed version of Google Chrome is 58.0.3029.110, even when using the path option to point it to version 64.0.3282.140

~ % gowitness single https://google.com --chrome-path="/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome"
FATA[2018-02-05 10:44:23] Chrome version is older than v60              chromeversion=58.0.3029.110

~ % /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --version
Google Chrome 64.0.3282.140

Even stops me from running commands like version

~ % gowitness version
FATA[2018-02-05 10:46:38] Chrome version is older than v60              chromeversion=58.0.3029.110

Describe the solution you'd like
Add HTTP and SOCKS5 proxy support screenshot functionality

Describe alternatives you've considered
Related to the old issue #21

Additional context
Seems like chromedp has this built in support already, just need to stub in the options and test in chrome/chrome.go

https://github.com/chromedp/examples/pull/20/files/0bfcebe721cd401786649f72d0b7926369348623#diff-a5b9dfd3fa592c284ba488a7c748679e

Current Behavior:
Take a list of 20k IP:PORT and use gowitness. From what i've seen the report output doesnt care to categories HTTP response codes. Meaning there is just a hoshposh of service responses mixed within the result. This can make it difficult to analyze even fairly small networks.

Expected Behavior
The output should be categorized by server response (at the very least).

i clone it, then i run go build then i try to use it , but nothing happen
what should i do next ?

thank you

./gowitness report generate -D test.db -d /tmp/test/asdfasdf/

will generate an html report and put it in the current working directory, and not place the report in the designated dir based on the cmdline. this is problematic for automation when screenshotting multiple domains using a wrapper script. Also it appears you cant give the report a name other than 'report-0.html', which is also problematic if you're screenshotting several dozen or several hundred domains and each domain needs to have its own report.

Getting this even with single flag
gowitness single --url=https://example.com

ERRO[2018-11-15 12:53:15] Failed to query url error="[Get https://example.com/?x=us: read tcp 192.168.31.153:37728->x.x.x.x:443: i/o timeout]" url="https://example.com." INFO[2018-11-15 12:53:15] Complete run-time=5.903457728s

Latest version with the same problem when hitting pages that are returning just a JSON string (for example).

(...)
ERRO[2020-01-25 12:57:55] Screenshot failed                             destination=https-REDACTED.com.png err="exit status 1" url="https://REDACTED/"
panic: runtime error: invalid memory address or nil pointer dereference
(...)

The page returned this when using curl:

> GET / HTTP/1.1
> Host: REDACTED.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json
< Client-IP: REDACTED
< GatewayMessage: error
< WWW-Authenticate: Bearer
< X-Backside-Transport: FAIL FAIL
< Connection: close
<
* TLSv1.2 (IN), TLS alert, close notify (256):
* Closing connection 1
* TLSv1.2 (OUT), TLS alert, close notify (256):
{ "ServiceStatus":{ "StatusType":"Error", "ServiceCode":"", "ServiceName":"
(REDACTED)
 "InnerException":"", "ServerName":"" }

Originally posted by @BBerastegui in #24 (comment)

Hi! So when I supply list of urls which don't have the http:// or https:// . I'm getting this error Skipping Invalid URL so does gowitness automatically prefix urls with http:// or https:// as mentioned in this issue #11 or do we have to supply an additional argument?

The command which I used: gowitness file -s ~/Desktop/urls.txt

Hello,

I made a few test, but this chinese websites don't work

./gowitness single --chrome-timeout 100 -u http://www.qry.com/Home.aspx 
ERRO[2019-02-24 15:55:32] Failed to query url                           error="[Get http://www.quippbattery.com/Home.aspx: read tcp 192.168.178.41:35752->222.76.212.44:80: i/o timeout]" url="http://www.quippbattery.com/Home.aspx"
INFO[2019-02-24 15:55:32] Complete                                      run-time=5.266110312s

I think the "Failed to query url" Error comes to early. " --chrome-timeout 100" has no effect.

Thank you, Best Regards
Simon

In future versions, would it be possible for the HTTP Title to be extracted when the GET response is parsed? That would be awesome.

Thanks,
Rhynorater

I guess it would be a good idea to breakdown report.html into different html files rather than display it at a single file.

From time to time, when scanning a large scope, the report.html becomes unresponsive due lack of memory (yeah! my laptop suck!).

So, I think this could be avoided by segmented report into, report.html, report_01.html, report_02.html etc. Maybe 20 screenshots in each file ?

I'll leave it up to you to decide! What you think of that idea ?

Once again, thanks a lot to come up with this awesome tool.

Cheers

Any plans to add support for RDP and VNC screenshots? There doesn't seem to be any tools out there that do this anymore and it would be nice if gowitness supported it.

Describe the bug
The proxy server -p and nmap port selector -p are the same flag. One of them will need to be changed, could just drop the shorthand nmap -p and leave --port, or assign a random letter to nmap's -p

To Reproduce

gowitness-2.1.0-linux-amd64 nmap -h
panic: unable to redefine 'p' shorthand in "nmap" flagset: it's already used for "port" flag

Version Information:

• gowitness: 2.1

Hi, I get this error message while running the script on a url file list.

FATA[2018-04-21 17:10:18] fork/exec /usr/bin/chromium: too many open files

I like to use firefox headless mode instead of chromium to taking screenshots. Firefox users can take screenshot using --screenshot flag via firefox binary. Check this resource to get more information:
Take a screenshot from the command line with headless

Describe the bug
Nmap files containing MAC addresses are included in the scan for URL's when parsing the nmap XML file

To Reproduce
Example NMAP File containing multiple addresses:

<address addr="192.168.1.1" addrtype="ipv4"/>
<address addr="00:00:00:00:00:00" addrtype="mac" vendor="FakeVendor"/>

Gowitness Result:

17 Sep 2020 10:50:00 ERR preflight failed for url error="Get \"https://00:00:00:00:00:00\": dial tcp: 
lookup 00:00:00:00:00:00: no such host"

Expected behavior
Just need to modify cmd/nmap.go to exclude an address type of 'mac'

Version Information:

• gowitness v2

Additional context
I'm 99% sure MAC addresses can't be used in a URL, but not 100% sure

Hello,

I try include the "--fullPage" flag for chrome, instead of resolution. But with no success.
This is very useful when you have pages with different lengths. Long product lists etc.

I would be very grateful for any help.

Best Regards
Askan

I'm currently running some rather ugly regex to get a list of vhosts out of the subject alternative name (SAN) list in a host's certificate file via way of nmap (using their default plugin set, one of which is ssl-certs). I then feed these into gowitness for screenshotting.

I noticed gowitness acknowledges DNS names from this list, but instead of browsing each one, it simply groups them under one IP. It would be nice to be able to instruct gowitness to screenshot every vhost/dns name it encounters for a given host in an nmap file as they may very well be entirely different websites.

It would be superb if gowitness was able to attempt screenshots for a list of default ports, on both http and https if given a text file (or stdin?) full of ips or hostnames.
as it stands now, gowitness needs to be told what protocol (http or https) and what port, but with enough threads and a low timeout, making attempts to stuff that isnt open/up goes pretty fast, so having a list of like half a dozen default ports to try (80, 8080, 443, 4433, 8888, 81, 90 etc) and to try both http and https for each as "default functionality" would dramatically increase gowitnesses portability!

Is your feature request related to a problem? Please describe.
I'm miss some screenshots with chrome headless

Describe the solution you'd like
I would like if chromium is available for screenshotting the sites.

Describe alternatives you've considered
Chromium

Additional context
I hope it will be in actions ASAP.

Pretty simple stuff: I couldn't find examples for file flag when checking the docs. Also, the info presented when running --help doesn't say much. Ended up checking the file.go to get the proper syntax and then came across this:

$ gowitness file -s ~/Desktop/urls
$ gowitness file --source ~/Desktop/urls --threads -2 

so maybe just add it directly to the main docs?

Thanks

I'm wondering if it possible to go troufht a subdomain with out the quote http://

Is your feature request related to a problem? Please describe.
When doing recon on domains containing specific words, i'm really interested in the registrant information.

Describe the solution you'd like
Having a button in the report to view whois data. As well as a button to view the HTTP headers. Something like how Aquatone generates reports.

Describe alternatives you've considered
Trying to have a seperate process fetch whois data for all the domains and edit the report template to search for the corresponding whois data.

such as https://www.csdn.net

This can be added with the -dump-dom Chrome flag: https://developers.google.com/web/updates/2017/04/headless-chrome#printing_the_dom

Imagine we have an nmap output file that gave us which ports are opened and which ones are closed. gowitness instead of go only for the opens goes for every possible port therefore it does not make any distinction between open or closed ports.

The expected behaviour would be to only enumerate the open ports

gowitness version: Latest, downloaded yesterday

Nmap command:
nmap -iL FILE -p80,8000 -T4 -oX OUTPUT --max-retries=2

Gowitness command:
gowitness nmap --nmap-file OUTPUT --threads 20

Hi, First of all great tool! 👍
I have recently created a subdomain enumeration tool in golang and wanted to integrate gowitness as a data analysis module to my tool. But there are no clear instructions on how to do so. Therefore, anyone wanting to integrate it would have to work it out themselves.
It would be awesome if there were some instructions detailing how to integrate it into your golang project.
Thanks.

gowitness constantly gives the following error after updating to 1.1.0

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x14093c6]

goroutine 5183 [running]:
github.com/sensepost/gowitness/utils.getPageTitle(0xc0001bee70, 0xc000ae4d98, 0x100e448)
	/Users/leonjza/Documents/git/gowitness/utils/processor.go:124 +0xa6
github.com/sensepost/gowitness/utils.getPageTitle(0xc0001bee00, 0xc000ae4de0, 0x138c945)
	/Users/leonjza/Documents/git/gowitness/utils/processor.go:128 +0x7a
github.com/sensepost/gowitness/utils.getPageTitle(0xc0001bed90, 0x0, 0x0)
	/Users/leonjza/Documents/git/gowitness/utils/processor.go:128 +0x7a
github.com/sensepost/gowitness/utils.getPageTitle(0xc0001becb0, 0xc000c0d0a0, 0xc0001becb0)
	/Users/leonjza/Documents/git/gowitness/utils/processor.go:128 +0x7a
github.com/sensepost/gowitness/utils.ProcessURL(0xc000121480, 0x1b62de0, 0x1b60bb0, 0x9)
	/Users/leonjza/Documents/git/gowitness/utils/processor.go:47 +0x51c
github.com/sensepost/gowitness/cmd.glob..func1.1(0xc00021e500, 0xc000241ac0, 0xc0002446c0, 0xc000121480)
	/Users/leonjza/Documents/git/gowitness/cmd/file.go:94 +0x8a
created by github.com/sensepost/gowitness/cmd.glob..func1
	/Users/leonjza/Documents/git/gowitness/cmd/file.go:90 +0x5ec

ERRO[2020-05-12 16:43:39] Screenshot failed destination=https-www.google.com.png err="exit status 1" url="https://www.google.com/"
ERRO[2020-05-12 16:43:39] Chrome process reported an error taking screenshot error="exit status 1" url="https://www.google.com/"

it is giving me an error every time I do a screenshot, I have chrome and chromium latest version installed, anyone knows whats wrong?

Hi @leonjza Its not an issue but a request to have a feature for importing url's based on particular status code via the report viewer. Like a feature to filter screenshots based on status code and then export the urls to a text file.
Also have an option to increase number of screenshot per page and next button on the top too.

Regards,

Describe the bug
No such file or directory while the file exist

To Reproduce
Steps to reproduce the behavior:

~/go/bin/gowitness file domain.txt                                                                                             10:22:42
FATA[2020-05-12 10:22:49] Unable to read source file                    error="open : no such file or directory" source=

Expected behavior
File to be read

Version Information:

• OS: Linux
• gowitness: latest version

Problem: User-Agent is always "Go-http-client/1.1"

Expected behavior: According to the binary's help documentation, the default UA is supposed to be "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.75 Safari/537.36", though, testing shows it is "Go-http-client/1.1" without using the --user-agent flag. Setting the --user-agent flag to something different , also results the same - "Go-http-client/1.1"

I noticed this due to many of the screenshots saying something along the lines of "prove you're not a bot".

Please let me know how to fix this, or if it is my mistake. Also, thank you for the excellent tool!

Would it work?

It would be useful if this tool supported a file input to provide a list of urls for screenshots.

gowitness should have the option of prepending http/s like eyewitness does.

Is your feature request related to a problem? Please describe.
Lot's of WAF screenshots

Describe the solution you'd like
accept cookies to send

I'm attempting to batch collect images from hundreds of websites. Using gowitness file with threads seems to work fine. However for more control I'd like to use gowitness single as it's part of a larger bash script. However running multiple gowitness single commands or even many gowitness single command consecutively and I'm get panic errors as shown below.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1357950]

goroutine 1 [running]:
github.com/tidwall/buntdb.(*Tx).lock(0xc0004f8440)
        /Users/austinginder/go/src/github.com/tidwall/buntdb/buntdb.go:1088 +0x30
github.com/tidwall/buntdb.(*DB).Begin(0x0, 0xc000126701, 0x115e9c5, 0xc0000881e0, 0xc0001420c0)
        /Users/austinginder/go/src/github.com/tidwall/buntdb/buntdb.go:1067 +0x61
github.com/tidwall/buntdb.(*DB).managed(0x0, 0x115c901, 0xc0001268b8, 0x0, 0x0)
        /Users/austinginder/go/src/github.com/tidwall/buntdb/buntdb.go:939 +0x40
github.com/tidwall/buntdb.(*DB).Update(0x0, 0xc0001268b8, 0x1, 0x1)
        /Users/austinginder/go/src/github.com/tidwall/buntdb/buntdb.go:984 +0x3a
github.com/sensepost/gowitness/storage.(*Storage).SetHTTPData(0x1929de8, 0xc00017e500)
        /Users/austinginder/go/src/github.com/sensepost/gowitness/storage/storage.go:53 +0x456
github.com/sensepost/gowitness/utils.ProcessURL(0xc00017e480, 0x192c0e0, 0x1929de8, 0x3c)
        /Users/austinginder/go/src/github.com/sensepost/gowitness/utils/processor.go:109 +0x1c99
github.com/sensepost/gowitness/cmd.glob..func5(0x19239c0, 0xc000088a20, 0x0, 0x6)
        /Users/austinginder/go/src/github.com/sensepost/gowitness/cmd/single.go:36 +0x8c
github.com/spf13/cobra.(*Command).execute(0x19239c0, 0xc000088960, 0x6, 0x6, 0x19239c0, 0xc000088960)
        /Users/austinginder/go/src/github.com/spf13/cobra/command.go:766 +0x2cc
github.com/spf13/cobra.(*Command).ExecuteC(0x1923500, 0xc000127f50, 0x14447ce, 0x1923500)
        /Users/austinginder/go/src/github.com/spf13/cobra/command.go:852 +0x2fd
github.com/spf13/cobra.(*Command).Execute(0x1923500, 0xc000092058, 0x0)
        /Users/austinginder/go/src/github.com/spf13/cobra/command.go:800 +0x2b
github.com/sensepost/gowitness/cmd.Execute()
        /Users/austinginder/go/src/github.com/sensepost/gowitness/cmd/root.go:93 +0x31
main.main()
        /Users/austinginder/go/src/github.com/sensepost/gowitness/main.go:6 +0x20

If helpful here is the format of the single command I'm using.

gowitness single --url="$url" --resolution="1200,750" --log-format=json --log-level=warn --timeout=60 --destination="/Users/austinginder/Desktop/Screenshots/"

Love the tool, but I'm running into an edge case that might only be solvable by a big hammer. Might be overkill for this project, but I wanted to propose just in case :-)

I'm trying to capture screenshots from go tool trace -http. Problem is, these pages can take up to 15 seconds to properly load. I've tried using --timeout 15 --chrome-time-budget 15 --chrome-timeout 15. None of those help - Chrome considered the page loaded, and AFAIK there is no method to simply say "please wait 10 seconds before you take the screenshot, even if the page reports it is loaded before then".

A google search leads me here, where a user has a similar issue. I looked into the two recommended solutions. Both internally utilize the Chrome DevTools Protocol, which allows you to connect to a headless instance and communicate with it. As would be expected, this allows a lot more power than passing flags, e.g. you wait for DOM events, or screenshot specific page elements , or otherwise engage with the page. The more developed project is https://github.com/puppeteer/puppeteer, which is a Google project for programmatically interacting with headless Chromium.

There is already a golang project that speaks the DevTools protocol - https://github.com/chromedp/chromedp. However, it seems focused on being a golang version of puppeteer - intended for programmatic usage. For example, here is their screenshot source code. This project is clean and simply focused on getting a screenshot.

Would you be open to PRs that attempt to utilize chromedp in some circumstances? Not quite sure the proper combination of flags to indicate when it should be used, that would need to be discussed. Perhaps some "wait X seconds after specified DOM event" approach?

Misc note: The API call for taking a screenshot using DevTools is in the 1.3 release candidate, whereas 1.2 is the current stable. Doubt it's an issue, as clearly puppeteer finds it OK to expose features from the RC protocol version, but thought it was worth mentioning regardless.

Timeout reached while waiting for screenshot to finish [31mdestination[0m=https-www.runoob.comechartsecharts-setup.html.png
...
command is gowitness single --url=....com
version:windows10
chrome-version:latest

When I move a report/db/screenshots around it breaks screenshot viewing because the report screenshot urls are absolute path'ed. This means I end up having to sed the leading path out. Can we get an option to add relative paths for screenshots?

I'm running the script using the file command, and have lots of URLs in my target file. The problem is when the URL is http://x.x.x.x and the response is a redirect to https://x.x.x.x, the script doesn't follow the redirect which results in lots of blank png files.