Inserted binary data differs from original about gopass HOT 7 CLOSED

Also tested with gopass 1.15.5 go1.20 linux amd64

from gopass.

If you're managing binary blobs the recommended way is to use gopass cat:

NAME:
   gopass cat - Decode and print content of a binary secret to stdout, or encode and insert from stdin

USAGE:
   gopass cat [command options] [secret]

DESCRIPTION:
   This command is similar to the way cat works on the command line. It can either be used to retrieve the decoded content of a secret similar to 'cat file' or vice versa to encode the content from STDIN to a secret.

OPTIONS:
   --help, -h  show help (default: false)

But it seems that this currently doesn't work neither:

cat test.blob | gopass cat blob
gopass cat blob
<empty output>

and looking into the secret it effectively didn't encode the data as it should have 😐
So yeah, we have a bug or two, there.
It might be related to the size of the data, that's a pretty big file at 1MB, but it should be well within what we can encrypt.

And wow, I just tried with insert and show, and yeah these files differ significantly, I'm not sure what's happening there. The ouput is only 1023873 bytes instead of 1024000, and when using gopass show -f -n test > test_out.blob3 it is still missing a few bytes at 1023988:

With xxd test_out.blob3 | head -n5:

00000000: a644 dcde 4080 b2a8 8b34 e17c 255d ac8e  [email protected].|%]..
00000010: bf76 0683 d821 56f6 6e5c 10b6 40ab 8c62  .v...!V.n\[email protected]
00000020: bc8f 7264 1f09 1fa6 c2d3 d67e ae7d 3feb  ..rd.......~.}?.
00000030: 2fb1 69de a601 d5a6 4e24 e109 a088 da54  /.i.....N$.....T
00000040: 40f3 5602 8220 f0de 6a44 45d6 e817 e710  @.V.. ..jDE.....

Which matches xxd test.blob | head -n5:

00000000: a644 dcde 4080 b2a8 8b34 e17c 255d ac8e  [email protected].|%]..
00000010: bf76 0683 d821 56f6 6e5c 10b6 40ab 8c62  .v...!V.n\[email protected]
00000020: bc8f 7264 1f09 1fa6 c2d3 d67e ae7d 3feb  ..rd.......~.}?.
00000030: 2fb1 69de a601 d5a6 4e24 e109 a088 da54  /.i.....N$.....T
00000040: 40f3 5602 8220 f0de 6a44 45d6 e817 e710  @.V.. ..jDE.....

But things go wrong somewhere in the middle: xxd test.blob | tail:

000f9fe0: 28cd 9323 c0db 1647 2749 b2b3 fb70 49a6  (..#...G'I...pI.
000f9ff0: 5008 d44d 6bcb 3cc2 63a8 895d 7e30 a5c2  P..Mk.<.c..]~0..

doesn't match the output of xxd test_out.blob3 | tail -n 2 (notice the offset, it's not even truncated at the end but somewhere in the middle):

000f9fe0: 7049 a650 08d4 4d6b cb3c c263 a889 5d7e  pI.P..Mk.<.c..]~
000f9ff0: 30a5 c20a                                0...

from gopass.

Thanks for the report and the investigation.

gopass cat should indeed work for binary files.
Looks like a bug.

from gopass.

It seems to me that a smaller binary-sice works.

--- tested on v1.15.6-3-g8c666a93 ---

$ dd if=/dev/random bs=1K count=1 of=workdir/bin
1+0 records in
1+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000266374 s, 3.8 MB/s
$ cat workdir/bin | gopass cat testing.bin      
$ gopass cat testing.bin > workdir/test         
$ shasum workdir/bin workdir/test               
b538947e4010c851d8295ebaaa66cfdaf563e072  workdir/bin
b538947e4010c851d8295ebaaa66cfdaf563e072  workdir/test

Currently I could not pin down the issue any further, but maybe this helps.

from gopass.

Thanks. I need to carve out some time to reproduce this. It should work for any size (up to filesystem / OS limits).
Since we don't do any chunking / stream processing RAM might be a limit, but since gopass is not designed to handle huge files I don't intend to change that.

from gopass.

The problem why gopass insert doesn't work is that we try to parse the input with a line scanner here: https://github.com/gopasspw/gopass/blob/master/pkg/gopass/secrets/akv.go#L37

This will at least add an extra \n to the input. So the checksums won't match. Usually a no-op for text, but a deal breaker for binary data. Now we have two issues:

1. Can we reliably make the AKV parser detect binary data and pass it through unaltered? Without breaking any existing use cases, of course.
2. Figuring out what's wrong with cat ... | gopass cat secret. That must work for sure.

from gopass.

For 2.: Turns out it's not writing but reading that's broken. Looking into a fix.

from gopass.