[BUG] Invalid JSON output for Content-Security-Policy finding including TABs (ascii 0x09) about testssl.sh HOT 14 CLOSED

PS: see also https://dev.testssl.sh

from testssl.sh.

from testssl.sh.

I was experimenting with sed on linux and freebsd but concluded that tr works better for both platforms

linux

$ python3 -c "print(''.join(map(chr, range(127))))" | tr '\000-\037' '?' | xxd
00000000: 3f3f 3f3f 3f3f 3f3f 3f3f 3f3f 3f3f 3f3f  ????????????????
00000010: 3f3f 3f3f 3f3f 3f3f 3f3f 3f3f 3f3f 3f3f  ????????????????
00000020: 2021 2223 2425 2627 2829 2a2b 2c2d 2e2f   !"#$%&'()*+,-./
00000030: 3031 3233 3435 3637 3839 3a3b 3c3d 3e3f  0123456789:;<=>?
00000040: 4041 4243 4445 4647 4849 4a4b 4c4d 4e4f  @ABCDEFGHIJKLMNO
00000050: 5051 5253 5455 5657 5859 5a5b 5c5d 5e5f  PQRSTUVWXYZ[\]^_
00000060: 6061 6263 6465 6667 6869 6a6b 6c6d 6e6f  `abcdefghijklmno
00000070: 7071 7273 7475 7677 7879 7a7b 7c7d 7e3f  pqrstuvwxyz{|}~?

freebsd

$ python3 -c "print(''.join(map(chr, range(127))))" | tr '\000-\037' '?' | xxd
00000000: 3f3f 3f3f 3f3f 3f3f 3f3f 3f3f 3f3f 3f3f  ????????????????
00000010: 3f3f 3f3f 3f3f 3f3f 3f3f 3f3f 3f3f 3f3f  ????????????????
00000020: 2021 2223 2425 2627 2829 2a2b 2c2d 2e2f   !"#$%&'()*+,-./
00000030: 3031 3233 3435 3637 3839 3a3b 3c3d 3e3f  0123456789:;<=>?
00000040: 4041 4243 4445 4647 4849 4a4b 4c4d 4e4f  @ABCDEFGHIJKLMNO
00000050: 5051 5253 5455 5657 5859 5a5b 5c5d 5e5f  PQRSTUVWXYZ[\]^_
00000060: 6061 6263 6465 6667 6869 6a6b 6c6d 6e6f  `abcdefghijklmno
00000070: 7071 7273 7475 7677 7879 7a7b 7c7d 7e3f  pqrstuvwxyz{|}~?

from testssl.sh.

Thanks a lot for reporting! Can reproduce this.

First thing popped up: Just scratching my head though how we should handle this as on one hand we shouldn´t pass certain characters 1:1 which may cause trouble in different file outputs. And OTOH we want the user to give a most real view of the headers.

from testssl.sh.

I think that correct JSON encoding actually preserves "most real view" of the data in context of respective data representation format. So, patching all places where raw data values are sent to JSON output should be treated manually as are in cases of CSV (safe_echo) or HTML (html_reserved). However in case of current bash implementation it could be somewhat harsh.

from testssl.sh.

For every output I believe there´s a problem with chars <= 0x20.

What I am currently struggling to understand: the findings in JSON are sanitized with printf -- ¨%b" . On the cmdline there was no issue.

from testssl.sh.

MVP script using printf produces string including ascii TAB, which is correct behavior documented in man
(also correct for terminal output)

%b ARGUMENT as a string with '' escapes interpreted, except that octal escapes are of the form \0 or \0NNN

$ cat >test.sh  <<__EOF__
printf -- "%b"  "a	b\tc"
__EOF__

$ bash test.sh | xxd
00000000: 6109 6209 63                             a.b.c

but it is not correct JSON, where scpecs requires all characters <= 0x1f to be escaped

9 String

A string is a sequence of Unicode code points wrapped with quotation marks (U+0022). All code points may
be placed within the quotation marks except for the code points that must be escaped: quotation mark
(U+0022), reverse solidus (U+005C), and the control characters U+0000 to U+001F

$ cat >test1.sh <<__EOF__
python3 -c 'import json; print(json.dumps("a	b"))'
__EOF__

$ bash test1.sh
"a\tb"

from testssl.sh.

yeah, that's the output with wget too. But python is not an option and yoiur example works for JSON only. I am sure we'll find a way.

from testssl.sh.

yeah, that's the output with wget too. But python is not an option and yoiur example works for JSON only. I am sure we'll find a way.

yeah, just to be on the same page:

• I've used python only to document correct expected behavior
• there's no single solution for multiple output formats i'm sure

I guess some awk or sed based function will eventually suffice

• https://stackoverflow.com/questions/10053678/escaping-characters-in-bash-for-json/74426351#74426351
  (I'd elaborate on this solution but only for proper subset of characters)

• custom

$ echo -e "a\tb" | sed 's/\t/\\t/g'
a\tb

from testssl.sh.

I've used python only to document correct expected behavior

I thought so, but just want to make sure.

there's no single solution for multiple output formats i'm sure

I guess some awk or sed based function will eventually suffice

2nd: probably. 1st; naah, I disagree. ;-) I currently believe a sanitizing function which removes the escape codes would be the best which will be called for all output formats.

from testssl.sh.

For JSON and HTML under GNU sed this e.g. seems to work:

sed  's/[\o00\o01\o02\o03\o04\o05\o06\o07\o10\o11\o12\o13\o14\o15\o16\o17\o20\o21\o22\o23\o27\o30\o31\o32\o33\o34\o35\o36\o37]//g'

For the .log file we would need to omit octal 33 (0x1b).

I am struggling a bit with BSD sed. It works with a single char like sed $'s/[\011]//g' some chars lead to the error unbalanced brackets ([])

Any clues?

from testssl.sh.

Thanks both!

I think we should settle on tr -d '\000-\011,\013-\037' . Works on Linux, MacOSX and OpenBSD.

The python generator is a neat thing to show what works.

\033 needs to be skipped for *log files.

from testssl.sh.

The terminal might be still a problem. Normally one tries to filter at input. The approach above does not suffice when looking at

testssl.sh --header dev.testssl.sh or even testssl.sh --color-0 --header dev.testssl.sh

from testssl.sh.

Thanks @bodik for reporting and thanks to the others help discussing.

I am closing this at the original issue (JSON) has been fixed and opened another issue for file/terminal output

from testssl.sh.