`std.compress.flate.decompress()` results in `error.EndOfStream` about zig HOT 7 CLOSED

Are you using websocket with no_context_takeover?
If that is negotiated between server and client then each message is independently compressed otherwise each message is block in 'endless' deflate stream. In that case you must use same compressor/decompressor for the entire stream.

That is not important for the first message but it will help understand why web sockets never sets bfinal bit.
Bfinal is first bit in first byte (those difference between 242 (not set) and 243 (set)). If it is set it defines following deflate block as last. Decompressor will not expect any new blocks after it decompresses that last block.

Deflate is bit stream (not a byte stream) it can finish block on any bit in the byte. The new block starts at next bit. To split deflate stream into bytes websocket per-message deflate inserts empty stored block at the end. That guarantees that we finish at the byte end. Stored block starts with 3 bits header (as any other block), but after that aligns to the byte boundary (as no other block). After header there are 4 bytes len/~len and then plain data. Empty stored block has 0x0 0x0 0xff 0xff for len/~len and no body.
Websocket rfc instructs compressor to finish with empty stored block, then remove last 4 bytes and that send to the wire. Decompressor should add 4 bytes 0x0 0x0 0xff 0xff and then decompress.

If we look at your first example: { 242, 72, 205, 201, 201, 87, 4, 0 } with infgen:

! infgen 3.2 output
!
fixed                   ! 01 0
literal 'H              ! 00011110
literal 'e              ! 10101001
literal 'l              ! 00111001
literal 'l              ! 00111001
literal 'o              ! 11111001
literal '!              ! 10001010
end                     ! 0000000
!
stored                  ! 00 0
                        ! 000
infgen warning: incomplete deflate data

We have non final type 1 (fixed) block and header of the stored block.

{ 242, 72, 205, 201, 201, 87, 4, 0, 0, 0, 0xff, 0xff } is much better because it leaves stream at end of stored block, ready to parse next block. Important if you will have next message with the same context in websocket stream.

! infgen 3.2 output
!
fixed                   ! 01 0
literal 'H              ! 00011110
literal 'e              ! 10101001
literal 'l              ! 00111001
literal 'l              ! 00111001
literal 'o              ! 11111001
literal '!              ! 10001010
end                     ! 0000000
!
stored                  ! 00 0
                        ! 000
end
!
infgen warning: incomplete deflate data

For decompressing websocket messages you should append 0 0 0xff 0xff to the end of each message, then use decompressor until you got EndOfStream error. Error means that decompressor has reached end of stream but the last block it decompressed didn't have bfinal bit set. Which in websocket per message deflated stream never will be set.

This block: { 243, 72, 205, 201, 201, 87, 112, 206, 201, 76, 205, 43, 81, 176, 210, 84, 4, 0 } has bfinal bit set (note last word in first line) and infgen does not complain about 'incomplete deflate data'. This is just example not relevant for the websocket.

! infgen 3.2 output
!
last                    ! 1
fixed                   ! 01
literal 'H              ! 00011110
literal 'e              ! 10101001
literal 'l              ! 00111001
literal 'l              ! 00111001
literal 'o              ! 11111001
literal '               ! 00001010
literal 'C              ! 11001110
literal 'l              ! 00111001
literal 'i              ! 10011001
literal 'e              ! 10101001
literal 'n              ! 01111001
literal 't              ! 00100101
literal '               ! 00001010
literal ':              ! 01010110
literal ')              ! 10011010
literal '!              ! 10001010
end                     ! 0000000
                        ! 000000

infgen is tool by Mark Adler, author of zlib

from zig.

If your are also using zig compressor for websocket, you will need something like:

• create compressor with some writer
• call compress, pass reader with plain data
• call flush - that will add empty stored block and write everything to the writer
• use data from writer, remove 4 bytes (0 0 ff ff) and send them on the wire
• repeat, call compress with another reader ...

from zig.

@squeek502

I experimented a bit and found the following:

• std.compress.flate.compress():
• • { 243, 72, 205, 201, 201, 87, 112, 206, 201, 76, 205, 43, 81, 176, 210, 84, 4, 0 }
• JavaScript WebSocket:
• • { 242, 72, 205, 201, 201, 87, 112, 206, 201, 76, 205, 43, 81, 176, 210, 84, 4, 0 }

JavaScript sends the same result but the first value is lower so I added a 1 and it works fine. Maybe you can explain what's wrong?

If I reduce the first value of the Zig compressed data, it also works fine for JavaScript decompression.

Decompress

Decompress data from other flate compressions with std.compress.flate.compress(). When I try to decompress the compressed data from JavaScript WebSocket it fails but not when I use Result[0] += 1.

Result: { 242, 72, 205, 201, 201, 87, 112, 206, 201, 76, 205, 43, 81, 176, 210, 84, 4, 0 }

• Result
• • error.EndOfStream
• Result[0] += 1
• • Hello Client :)!

Compress

Compress data with std.compress.flate.compress() and decompress with other flate decompressions. Decompressing with JavaScript WebSocket works fine even when I used Result[0] -= 1.

Result: { 243, 72, 205, 201, 201, 87, 112, 206, 201, 76, 205, 43, 81, 176, 210, 84, 4, 0 }

• Result
• • Hello Client :)!
• Result[0] -= 1
• • Hello Client :)!

I'm not sure if this is a Zig problem, but it looks like it to me. Apparently the first byte in the slice is too high by 1 when compressing and too low when decompressing.

from zig.

This deflate reference implementation also fails with error code 2 for this input:

2:  available inflate data did not terminate

Could you provide some more information about why this input should not be failing?

For future reference/ease-of-use, here's the input data as a Zig string:

"\xf2H\xcd\xc9\xc9W\x04\x00"

from zig.

@ianic

Wow! I didn't find anything about this... I'm sorry

Many thanks for your help

from zig.

Many thanks to @ianic for his really informative and helpful answer.

from zig.

@ThisAccountHasBeenSuspended Can you create some full working example?

from zig.