./util/test should be updated to add a function that tests the validity/soundness of twa's CSV output.

Something like csvlint would probably be appropriate.

#41 again:

You should use a custom user agent string that somewhat identifies the toolkit, for example twa/${TWA_VERSION} +https://github.com/trailofbits/twa

This can potentially get around the problem of some sites blocking generic curl user agents. It's also nice in general to tell what the source of automated requests is

We should use GitHub Actions to test twa, instead of Travis.

Should be fairly similar to the SCM checks.

See: https://twitter.com/svblxyz/status/1045013939904532482

nice little script BTW!

twa complains on my site about serving things like SCM and build files:

FAIL(example.com): TWA-0401: SCM repository being served at: http://example.com/.git/HEAD
FAIL(example.com): TWA-0407: Package management file being served at: http://example.com/.npmrc
FAIL(example.com): TWA-0409: Build file being served at: http://example.com/Makefile

My site doesn't actually serve content there. It sends a 301 back to the root of the site. It would be nice if twa knew that this wasn't actually serving the forbidden content that it's testing for.

This should use GitHub's dedicated workflows, rather than the stuff I've hacked together.

So - it looks like http/2 servers are giving false positives; I am seeing FAIL on headers that I know are sent (via inspection with curl -vvv) and that other tools (I tried https://tools.geekflare.com/tools/x-frame-options-test and it came back ok). Seen with Flask behind an AWS ALB.

Copy of the headers as see from curl -vvv:

< HTTP/2 200
< date: Mon, 17 Sep 2018 22:52:13 GMT
< content-type: text/html; charset=utf-8
< content-length: 2103
< x-frame-options: SAMEORIGIN
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< referrer-policy: no-referrer
< server: Werkzeug/0.12.2 Python/3.6.5
<

We should check for the Expect-CT header and come up with some best practices for it.

References:

• MDN: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Expect-CT

Per MDN: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permissions-Policy

Looks like this got renamed a while back; we should probably flag a MEH is a site reports Feature-Policy.

Looping over a list where one of the domains is not responsive/other will result in the script hanging due to curl not using a timeout.

You can use --max-connect or --max-time with a timeout value. maybe pass the value as a flag?

There are at least two more security relevant headers that we should test for and produce judgements on:

• Cross-Origin-Opener-Policy
• Cross-Origin-Embedder-Policy

More resources:

• https://hacks.mozilla.org/2020/07/safely-reviving-shared-memory/
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cross-Origin-Opener-Policy
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cross-Origin-Embedder-Policy

Similar to #56: Cookie prefixes are another proposed enhancement to cookie security. We should add appropriate checks for __Host and __Secure.

Resources:

• IETF draft: https://tools.ietf.org/html/draft-ietf-httpbis-cookie-prefixes-00
• Tough Cookies: https://scotthelme.co.uk/tough-cookies/

See

It might be interesting to add some Web Application Firewall detection techniques. I don't know much about WAFs, but it looks like there are some common oracles:

• Known cookies
• Known weird HTTP codes (999 No Hacking)
• Known HTML responses

Some potential resources:

• https://www.owasp.org/images/b/bf/OWASP_Stammtisch_Frankfurt_WAF_Profiling_and_Evasion.pdf
• https://www.securitynewspaper.com/2018/12/04/detect-web-application-firewall-waf-before-you-attack/ (looks like there's an nmap script for WAF detection)

It'd be nice if the various test result functions (PASS, FAIL, etc) could emit CSV when requested.

For example, when requested, FAIL would go from this:

FAIL(example.com): No CAA records found

to this:

fail,example.com,No CAA records found

In terms of requesting CSV output, either a flag (-c?) or an environment variable (TWA_FORMAT=csv?) would be okay (or both?).

These can take quite a while (due to timeouts) and aren't always relevant, so a user should be allowed to disable them with a command-line option.

Something similar to what we do with testssl would probably be best.

There are lots of other common development ports that we should check for, including (but not limited to):

• 5000 (Flask and Kestrel's default port)
• 9200 (Elasticsearch's REST API)

I think this happens when querying a domain that has CNAME and/or default subdomain records. In those cases, dig +short caa <domain> returns some lines that just have the associated records' value. For example:
$ dig +short caa example.com www.example.com. host1.example.com.

In this case default subdomain is www and there is a CNAME record pointing that at an actual host, but there is no CAA record for any of these.

twa is trying to treat these lines as CAA records:
UNK(example.com): Malformed CAA record? (flag=www.example.com., tag=, value=) UNK(example.com): Malformed CAA record? (flag=host1.example.com., tag=, value=)

Each of these codes needs a wiki page explaining why it exists and how to remedy it. I've gotten the ball rolling with a page for TWA-0001 -- use it as a template for new pages.

• TWA-0001 "Expected port 443 to be open, but it isn't"
• TWA-0101 "HTTP redirects to HTTPS using a 302"
• TWA-0102 "HTTP redirects to HTTP (not secure)"
• TWA-0103 "HTTP doesn't redirect at all"
• TWA-0201 "Skipping security checks due to no secure channel"
• TWA-0202 "Strict-Transport-Security max-age is less than 6 months"
• TWA-0203 "Strict-Transport-Security, but no includeSubDomains"
• TWA-0204 "Strict-Transport-Security, but no preload"
• TWA-0205 "Strict-Transport-Security missing"
• TWA-0206 "X-Frame-Options is 'sameorigin', consider 'deny'"
• TWA-0207 "X-Frame-Options is 'allow-from', consider 'deny' or 'none'"
• TWA-0208 "X-Frame-Options missing"
• TWA-0209 "X-Content-Type-Options missing"
• TWA-0210 "X-XSS-Protection is '0'; XSS filtering disabled"
• TWA-0211 "X-XSS-Protection sanitizes but doesn't block, consider mode=block?"
• TWA-0212 "X-XSS-Protection missing"
• TWA-0213 "Referrer-Policy specifies '${rp}', consider 'no-referrer'?"
• TWA-0214 "Referrer-Policy missing"
• TWA-0215 "Content-Security-Policy 'default-src' is '${csp_default_src}'"
• TWA-0216 "Content-Security-Policy 'default-src' is missing"
• TWA-0217 "Content-Security-Policy has one or more 'unsafe-inline' policies"
• TWA-0218 "Content-Security-Policy has one or more 'unsafe-eval' policies"
• TWA-0219 "Content-Security-Policy missing"
• TWA-0220 "Feature-Policy missing"
• TWA-0301 "Site sends 'Server' with what looks like a version tag: ${server}"
• TWA-0302 "Site sends a long 'Server', probably disclosing version info: ${server}"
• TWA-0303 "Site sends '${badheader}, probably disclosing version info: ${content}'"
• TWA-0401 "SCM repository being served at: ${url}"
• TWA-0402 "Possible SCM repository being served (maybe protected?) at: ${url}"
• TWA-0403 "Environment file being served at: ${url}"
• TWA-0404 "Possible environment file being served (maybe protected?) at: ${url}"
• TWA-0405 "Config file being served at: ${url}"
• TWA-0406 "Possible config file being served (maybe protected?) at: ${url}"
• TWA-0407 "Package management file being served at: ${url}"
• TWA-0408 "Possible package management file being served (maybe protected?) at: ${url}"
• TWA-0409 "Build file being served at: ${url}"
• TWA-0410 "Possible build file being served (maybe protected?) at: ${url}"
• TWA-0501 "No robots file found at: ${domain}/robots.txt"
• TWA-0502 "robots.txt lists what looks like an admin panel"
• TWA-0503 "robots.txt lists what looks like CGI scripts"
• TWA-0504 "No security file found at: ${domain}/.well-known/security.txt"
• TWA-0601 "No CAA records found"
• TWA-0602 "Domain doesn't specify any valid issuers"
• TWA-0603 "Domain explicitly disallows all issuers"
• TWA-0604 "Domain doesn't specify any violation reporting endpoints"
• TWA-0701 "Domain is listening on a development/backend port: ${dev_port}"
• TWA-0801 "cookie '${cookie_name}' has 'secure' but no 'httponly' flag"
• TWA-0802 "cookie '${cookie_name}' has no 'secure' flag"

xfo=${xfo,,} on line 212 of twa seems to be causing the error:
./twa: line 212: ${xfo,,}: bad substitution

Changing the string to lowercase via,, parameter expansion was introduced in Bash version 4 [1] and OSX 10.13 ships with Bash version 3.2. Is there any chance of changing this to a tr or some other method of lower casing characters?

[1] - http://wiki.bash-hackers.org/scripting/bashchanges - listed as 'case modification operators for expansions'

Thanks for this wonderful tool!
Works like a charm, but.

I am using a domain like some.mysite.com for my web app, twa continues to add www. prefix to it, and surely fails.

So, it really affects the final score and CI process.
Can we please have an option to skip www. checks for subdomains?

An idea to have a new flag to display certain audit conditions.
Normal example run.

$ twa google.com

FAIL(google.com): TWA-0102: HTTP redirects to HTTP (not secure)
FAIL(google.com): TWA-0205: Strict-Transport-Security missing
MEH(google.com): TWA-0206: X-Frame-Options is 'sameorigin', consider 'deny'
FAIL(google.com): TWA-0209: X-Content-Type-Options missing
FAIL(google.com): TWA-0210: X-XSS-Protection is '0'; XSS filtering disabled
FAIL(google.com): TWA-0214: Referrer-Policy missing
FAIL(google.com): TWA-0219: Content-Security-Policy missing
FAIL(google.com): TWA-0220: Feature-Policy missing
PASS(google.com): Site sends 'Server', but probably only a vendor ID: gws
PASS(google.com): Site doesn't send 'X-Powered-By'
PASS(google.com): Site doesn't send 'Via'
(OMITTED)

Example to show only FAIL conditions. Note, both MEH and PASS types have been removed from this request.

$ twa -t FAIL google.com

FAIL(google.com): TWA-0102: HTTP redirects to HTTP (not secure)
FAIL(google.com): TWA-0205: Strict-Transport-Security missing
FAIL(google.com): TWA-0209: X-Content-Type-Options missing
FAIL(google.com): TWA-0210: X-XSS-Protection is '0'; XSS filtering disabled
FAIL(google.com): TWA-0214: Referrer-Policy missing
FAIL(google.com): TWA-0219: Content-Security-Policy missing
FAIL(google.com): TWA-0220: Feature-Policy missing
(OMITTED)

Example to show only PASS conditions.

$ twa -t PASS google.com

PASS(google.com): Site sends 'Server', but probably only a vendor ID: gws
PASS(google.com): Site doesn't send 'X-Powered-By'
PASS(google.com): Site doesn't send 'Via'
(OMITTED)

Maybe this idea is a bit overkill since you can simply use grep to achieve the same result.

Example to show only FAIL conditions using grep.

$ twa google.com | grep -e FAIL

FAIL(google.com): TWA-0102: HTTP redirects to HTTP (not secure)
FAIL(google.com): TWA-0205: Strict-Transport-Security missing
FAIL(google.com): TWA-0209: X-Content-Type-Options missing
FAIL(google.com): TWA-0210: X-XSS-Protection is '0'; XSS filtering disabled
FAIL(google.com): TWA-0214: Referrer-Policy missing
FAIL(google.com): TWA-0219: Content-Security-Policy missing
FAIL(google.com): TWA-0220: Feature-Policy missing
(OMITTED)

I'm open to discussion for constructive criticism or alternative suggestions to the idea.

I'm curious why they are considered failures, if google is not doing them? Clearly, google must have a reason for failing these tests. thoughts?

I think it'd be useful if the program had a version number.

The UI of the main (index.html) web page release looks very undecorated plain. I can revamp the UI using my frontend dev skills without using any external CSS or JS library, only plain HTML, CSS, and JS. Do let me know if I can start working.

Thanks!

308 is a valid permanent redirect code like 301, so twa should handle it correctly. Right now, it results in an UNK:

UNK(redacted): HTTP sends an HTTPS location but with a weird response code: 308

See line line 212: ${xfo,,}: bad substitution, something seems to break here.

./twa beta.knubbelmac.de Connection to beta.knubbelmac.de port 443 [tcp/https] succeeded! MEH(beta.knubbelmac.de): HTTP redirects to HTTPS using a 302 PASS(beta.knubbelmac.de): max-age is at least 6 months MEH(beta.knubbelmac.de): Strict-Transport-Security, but no includeSubDomains MEH(beta.knubbelmac.de): Strict-Transport-Security, but no preload ./twa: line 212: ${xfo,,}: bad substitution PASS(beta.knubbelmac.de): Site sends 'Server', but probably only a vendor ID: Apache PASS(beta.knubbelmac.de): Site doesn't send 'X-Powered-By' PASS(beta.knubbelmac.de): Site doesn't send 'Via' PASS(beta.knubbelmac.de): Site doesn't send 'X-AspNet-Version' PASS(beta.knubbelmac.de): Site doesn't send 'X-AspNetMvc-Version' PASS(beta.knubbelmac.de): No SCM repository at: http://beta.knubbelmac.de/.git/HEAD PASS(beta.knubbelmac.de): No SCM repository at: http://beta.knubbelmac.de/.hg/store/00manifest.i PASS(beta.knubbelmac.de): No SCM repository at: http://beta.knubbelmac.de/.svn/entries

According to RFC 6844 a certificate authority should search up the DNS hierarchy for a CAA record:

https://tools.ietf.org/html/rfc6844#page-7

Currently twa only checks the actual domain name.

Reproducing

$ twa
Usage: twa [-w] <domain>

$ twa --help
Usage: twa [-csvw] <domain>

What I expect

Both outputs are the same

What actually happened

twa doesn't show all possible flags

We should identify some best practices for these and add corresponding checks.

As an extension of #21, we could look for the following patterns in a site's robots.txt:

• /cgi-bin/ - Probably indicates a bunch of old junk, potentially with exposed and/or guessable names.
• wp-admin, phpMyAdmin - Probably indicates an admin panel (do people put this in robots.txt?).

These aren't security issues per se, but they probably fall under "huh, I didn't realize that was exposed" and probably deserve a MEH if found.

We currently check for secure and HttpOnly on cookies; we should also check the SameSite value and emit appropriate results.

Possible results:

• PASS for SameSite=Strict
• MEH for SameSite=Lax
• FAIL for SameSite=None

The absence of SameSite currently implies different things on different browsers, so its absence from a cookie should probably be a MEH for now.

Resources:

• IETF draft: https://tools.ietf.org/html/draft-west-first-party-cookies-07
• CSRF is (really) dead: https://scotthelme.co.uk/csrf-is-really-dead/

Using twa at commit 30e704b I get an incorrect "MEH" rating on the Strict-Transport-Security test.
However I configured my site to send the includeSubdomains; preload; options.

$ curl -I https://wikipiedi.it
[...]
Strict-Transport-Security: max-age=63072000; includeSubdomains; preload;
[...]

But:

$ ./twa wikipiedi.it
Connection to wikipiedi.it 443 port [tcp/https] succeeded!
PASS(wikipiedi.it): HTTP redirects to HTTPS using a 301
PASS(wikipiedi.it): max-age is at least 6 months
MEH(wikipiedi.it): Strict-Transport-Security, but no includeSubDomains
MEH(wikipiedi.it): Strict-Transport-Security, but no preload

line 212 xfo=${xfo,,} raises a bad substitution error.

It can be commented out.

Also, line 318,[[ "${BASH_VERSINFO[0]}" -ge 4 ]] || die "Expected GNU Bash 4.0 or later, got ${BASH_VERSION}"

should throw a warning, rather than doe. Script works on much earlier BBASH versions.

It'd be nice to have a check that tests for the presence of /.well-known/security.txt.

The absence of security.txt should probably be a MEH rather than a FAIL, just like the robots.txt check.

Reference: https://securitytxt.org/

.well-known (RFC) is becoming an increasingly popular destination for stashing site-wide metadata. Some of that metadata is relevant to site security or may unintentionally leak information, so we should scan it.

Some starting points:

• Presence of/interesting things in an MTA-STS policy (RFC)
  • This might be hampered by the fact that the RFC requires this policy to be hosted on a separate subdomain, e.g. mta-sts.example.com/.well-known/mta-sts.txt.
• Asset links: https://developers.google.com/digital-asset-links/v1/getting-started
• A number of different things on this list: https://en.wikipedia.org/wiki/List_of_/.well-known/_services_offered_by_webservers

If the server sends us one or more cookies:

• Each cookie should have the secure flag, to prevent transmission over HTTP. If any cookies are missing the secure flag, it's a FAIL.
  • Rationale: There's absolutely no reason to be sending cookies over plain old HTTP in 2018, so there's absolutely no reason to not pass the secure flag.
• Each cookie should have the httponly flag, to prevent client-side access. Each cookie that's missing an httponly should receive its own MEH.
  • Rationale: There are some legitimate use cases for client-side cookie access, but not many.

This might be outside the scope of twa, but it'd be informative if twa provided information about open, filtered, or closed ports on the server

Here's a list of potential ports

• 3306 (mysql)
• 9000 (default php-fpm port)
• 11211 (memcached)
• 8080 (prod shouldn't have dev environments)

That's as far as I can think of right now. But there's certainly more

twa should provide easily searchable numeric codes for its negative results (FATAL, FAIL, and MEH), similar to Shellcheck and other linting tools. GitHub's wiki feature could be leveraged to provide detailed explanations for each code, a la RuboCop.

For example, TWA-101 could be negative result 1 for stage 1: HTTP redirecting to HTTPS via a 302, instead of a more permanent 301.

Script is awesome, but i want to contribute it, and still dont understand how 😂

If you can, please, do it)
Btw, do you plan working with only shell script, or it would be more universal project with another code? (Python, for example)

Just making this to remind myself (unless @lojikil feels like taking a stab first).

Source: https://github.com/lojikil/lovetz/

Hi @woodruffw ,

First things first - thank you for this little pretty thing :) It's very useful tool and I really like it since the first time I've used it.

While I had no issues on Ubuntu 16.04 LTS, when I've tried to use twa on macOS, I run into this error:

$ ./twa google.com
Connection to google.com port 443 [tcp/https] succeeded!
FAIL(google.com): HTTP redirects to HTTP (not secure)
FAIL(google.com): Strict-Transport-Security missing
./twa: line 212: ${xfo,,}: bad substitution
PASS(google.com): Site doesn't send 'Server' header
PASS(google.com): Site doesn't send 'X-Powered-By'
PASS(google.com): Site doesn't send 'Via'
PASS(google.com): Site doesn't send 'X-AspNet-Version'
PASS(google.com): Site doesn't send 'X-AspNetMvc-Version'
PASS(google.com): No SCM repository at: http://google.com/.git/HEAD
PASS(google.com): No SCM repository at: http://google.com/.hg/store/00manifest.i
PASS(google.com): No SCM repository at: http://google.com/.svn/entries

Quick source code audit reveals three other places where this operator is used (in lines 231, 241 and 258).

I did some further investigation and found that as macOS uses older version of Bash ,, operator is not implemented, bc it was introduced in Bash ver. 4 (source: https://stackoverflow.com/questions/47815637/getting-bad-substitution-error-with-a-shell-script-on-a-mac)

Unfortunately, changing shell to sh (as mentioned in the thread on SO) does not resolve the problem, causing different syntax errors here and there.

twa still works despite of this issue, however an audit result for some headers are not present (eg. X-Frame-Options, X-XSS-Protection)

Any sugestions how ,, operator could be substituted with something Bash <4 compatible?

Thanks!

Cheers,

bl4de

testssl.sh is probably a good choice for integration with twa for SSL auditing, since it's a single file and doesn't have any non-system dependencies. We should only run it if it's present, emitting an UNK and continuing on if it isn't.

Source: https://github.com/drwetter/testssl.sh

In addition to the current SCM and docker checks, the following files could and should be checked for:

• Common config file names: config.xml, config.json, config.ya?ml, config.ini, config.cfg, etc
• Package manager files/artifacts: .npmrc, Gemfile, Gemfile.lock, Rakefile, setup.py, requirements.txt, etc
• Build files: Makefile, CMakeLists.txt