Hello,

The GitHub Security Lab team has found a potential vulnerability in your project. Please create a Security Advisory and invite me in to further disclose and discuss the vulnerability details and potential fix. Alternatively, please add a Security Policy containing a security email address to send the details to.

If you prefer to contact us by email, please reach out to [email protected] with reference to GHSL-2022-027.

Thank you,
Agustin Gianni
GitHub Security Lab

Compressing strings works perfectly with lz4. But I'm having trouble compressing images.

I've uploaded the image using CarrierWave gem. Here's how I display the image:

<%= image_tag profile.avatar_url %>

But when I try to compress using: <% comp = LZ4::compress(profile.avatar) %>,
it throws this error, " undefined method `bytesize' for #AvatarUploader:0x007fa1aaec49f8 ".

I've posted this on StackOverflow but no one has gotten back to me in the past week. I'd be happy if you could help me out on this! :)

https://github.com/komiya-atsushi/lz4-ruby#todo の『Support streaming methods.』についてですが、こちらでは生の LZ4 データを扱えるものが動作するものとして出来ており、手元には lz4cli で扱っている lz4 ストリームデータの入出力にも対応したものも出来ています (legacy stream は入力のみです)。

しかしながら実装されている API には差異が大きく、どうしたものかと悩んだまま pullreq を躊躇してしまっています 6(^ ^;

komiya-atsushi さんは raw stream API に対してどのように考えておられるのか、ご意見を頂きたいと思います。

dearblue による raw stream を実装したコードは https://github.com/dearblue/lz4-ruby/blob/support-rawstream/ext/lz4ruby/lz4ruby.c#LC238 以降にあります (2月中にコミットして、そのまま停滞しているものです)。

現段階における API は次のようになっています (いろいろと端折っていたりクラス・メソッド名がアレな感じだったりしますが):

• LZ4.raw_stream_encode : LZ4Internal::RawStreamEncoder.new の呼び出しで、圧縮器の生成
• LZ4.raw_stream_decode : LZ4Internal::RawStreamDecoder.new の呼び出しで、伸張器の生成
• LZ4Internal::RawStreamEncoder#update / LZ4Internal::RawStreamDecoder#update : LZ4.raw_compress / LZ4.raw_uncompress のストリーム圧縮・伸張処理版で、LZ4.raw_compress / LZ4.raw_uncompress とメソッド引数と戻り値が同じ
• LZ4Internal::RawStreamEncoder#reset / LZ4Internal::RawStreamDecoder#reset : 圧縮器・伸張器を使いまわせるように再初期化

生のストリーム処理の流れとしては:

1. LZ4.raw_stream_encode または LZ4.raw_stream_decode で生のストリーム圧縮器・伸張器を生成、取得する
2. 得た圧縮器・伸張器の #update を必要なだけ繰り返して呼び出す
3. 圧縮器・伸張器は使い終わったら使いっぱなしで、GC によって回収される (圧縮器の作業バッファは、ブロックサイズの分だけ確保する必要があります。これを即時に開放する機能があるのが望ましいと考えますが、実装には反映されていません)

となっています。

簡素な利用例については https://github.com/dearblue/lz4-ruby/blob/support-rawstream/lib/lz4-ruby.rb#LC54 のコメント中にあります。

あるいは独自リリースしてしまってもよろしいでしょうか。

よろしくお願いします。

I tried using lz4-ruby to implement compression in my Cassandra driver, cql-rb. Cassandra supports LZ4 compression of requests and responses, but lz4-ruby can't decompress the messages coming from Cassandra because it expects the message to have a header.

Is this header a standard LZ4 thing, or is it something you've added just to be able to use the faster decompression methods?

Hi Komiya

There is a new security fix for 32-bits system available at :
https://github.com/Cyan4973/lz4/releases/latest

The issue affects programs accepting very large data blocks in 32-bits mode.

As a security measure, all binders are invited to upgrade their underlying C source code to latest version.

Regards

Using lz4-ruby 0.3.1:

input = 'Ķ' * 100
output = LZ4.uncompress(LZ4.compress(input))
output.force_encoding(Encoding::UTF_8)
input.should == output # false

The exact characters don't matter, it just seems to matter that they are multibyte. Is there a String#size call somewhere that should have been a String#bytesize?

I saw in the source that ::uncompress is deprecated.

Hello, I haven't been able to decompress a .jsonlz4 file with this gem as it says my file is corrupt. I'm able to use a Python script on it successfully though.

require 'lz4-ruby'

filepath = "recovery.jsonlz4"
File.exists?(filepath) # true

file = File.read(filepath)
LZ4::uncompress(file)
# .rvm/gems/ruby-3.0.2/gems/lz4-ruby-0.3.3/lib/lz4-ruby.rb:28:in `uncompress': Compressed data is maybe corrupted. (LZ4Internal::Error)

# Python script removes the first 8 characters, so I also tried that
LZ4::uncompress(file[8..])
# .rvm/gems/ruby-3.0.2/gems/lz4-ruby-0.3.3/lib/lz4-ruby.rb:28:in `uncompress': Compressed data is maybe corrupted. (LZ4Internal::Error)

If this is not supported, that's fine. Wanted to make sure I'm not doing something wrong. I noticed streaming methods is mentioned as a to-do on the README which might be related.

Note: This filetype is used by Firefox for profile files.

Here is the following commands in irb that reproduce this issue. Why does my UTF8 encoded string turn into an ASCII string? When I uncompress it, it will remain ASCII. But, that is not my original string. Because of this conversion that takes place I actually lose some of my UTF8 data.

1.9.3-p484 :003 > t = "teststring"
=> "teststring"
1.9.3-p484 :004 > t.encoding
=> #Encoding:UTF-8
1.9.3-p484 :005 > require 'lz4-ruby'
=> true
1.9.3-p484 :006 > result = LZ4::compress t
=> "\n\xA0teststring"
1.9.3-p484 :007 > result.encoding
=> #Encoding:ASCII-8BIT

I'm having trouble decompressing (compressed with lz4-ruby) in the browser using lz4js. It seems lz4-ruby does not frame the resulting compressed bytes into frames.

lz4-ruby:

$ ruby -rlz4-ruby -e 'print LZ4.compress("hello, world")' | hexdump -C
00000000  0c c0 68 65 6c 6c 6f 2c  20 77 6f 72 6c 64        |..hello, world|
0000000e

Using the command lz4c:

$ echo -n 'hello, world' | lz4c  | hexdump -C
00000000  04 22 4d 18 64 40 a7 0c  00 00 80 68 65 6c 6c 6f  |."[email protected]|
00000010  2c 20 77 6f 72 6c 64 00  00 00 00 d7 ff a5 4f     |, world.......O|
0000001f

Am I doing something wrong?