Giter VIP home page Giter VIP logo

Comments (21)

pete4abw avatar pete4abw commented on June 5, 2024

Fixed. All branches.

from lrzip-next.

asdfugil avatar asdfugil commented on June 5, 2024

The fix is not proper because write() more than 0x7fffffff on macOS would always fail with errno 22.
So if nmemb is larger than 0x7fffffff, it should be set to 0x7fffffff before trying to write()

from lrzip-next.

pete4abw avatar pete4abw commented on June 5, 2024

The fix is not proper because write() more than 0x7fffffff on macOS would always fail with errno 22. So if nmemb is larger than 0x7fffffff, it should be set to 0x7fffffff before trying to write()

Would you mind taking a look at the code and suggest a better fix? I will reopen. Please provide additional details as you can. Thank you

from lrzip-next.

asdfugil avatar asdfugil commented on June 5, 2024

Limit the maximum size of nmemb to 0x7ffff000, the rest of the code in write_fdout will treat it as a partial write, so it is okay.

bool write_fdout(rzip_control *control, void *buf, i64 len)
{
        uchar *offset_buf = buf;
        ssize_t ret, nmemb;

        while (len > 0) {
                nmemb = len;
                if (nmemb > 0x7ffff000) nmemb = 0x7ffff000;
                ret = write(control->fd_out, offset_buf, (size_t)nmemb);
                /* error if ret == -1 only. Otherwise, buffer not wholly written */
                if (unlikely(ret == -1))        /* error, not underflow */
                        fatal("Failed to write %'"PRId64" bytes to fd_out in write_fdout\n", nmemb);
                len -= ret;
                offset_buf += ret;
        }
        return true;
}

According to POSIX.1, if count is greater than SSIZE_MAX the result is implementation-defined.
getconf SSIZE_MAX reports 9223372036854775807 on macOS, so that value can't be used either

So in order to be absolutely safe, the maximum amount of data that can be written at once would only be 32767 bytes.

from lrzip-next.

pete4abw avatar pete4abw commented on June 5, 2024

Limit the maximum size of nmemb to 0x7ffff000, the rest of the code in write_fdout will treat it as a partial write, so it is okay.
snip...
According to POSIX.1, if count is greater than SSIZE_MAX the result is implementation-defined. getconf SSIZE_MAX reports 9223372036854775807 on macOS, so that value can't be used either

So in order to be absolutely safe, the maximum amount of data that can be written at once would only be 32767 bytes.

This is standard for the write function. From the manpage:

On Linux, write() (and similar system calls) will transfer at most 0x7ffff000 (2,147,479,552) bytes, returning the number of bytes actually transferred.

Since I don't have a MAC to test this on, please run the unmodified program with -vv option and let's see what it's doing. Unless the write() function is not working as advertised, I don't see the error. Thank you.

from lrzip-next.

asdfugil avatar asdfugil commented on June 5, 2024

Output:
src/lrzip-next -vvd ~/Downloads/kc.tar.lrz

The following options are in effect for this DECOMPRESSION.
Threading is ENABLED. Number of CPUs detected: 10
Detected 17,179,869,184 bytes ram
Nice Value: 19
Show Progress
Max Verbose
Temporary Directory set as: /var/folders/v1/b2rv5mk51zl9280bvywyp_dh0000z8/T/
Output filename is: /Users/nick/Downloads/kc.tar
Malloced 5,726,617,600 for tmp_outbuf
Unknown hash, falling back to CRC
Not performing hash check
CRC32 being used for integrity testing.
Validating file for consistency...
Unknown hash, falling back to CRC
[OK]
Detected lrzip version 0.6 file.
Decompressing...
Reading chunk_bytes at 24
Expected size: 21,072,936,960
Chunk byte width: 5
Reading eof flag at 25
EOF: 1
Reading expected chunksize at 26
Chunk size: 21,072,936,960
Reading stream 0 header at 32
Reading stream 1 header at 48
Reading ucomp header at 2,763,109,333
Fill_buffer stream 0 c_len 26,616,795 u_len 65,282,515 last_head 0
Starting thread 0 to decompress 26,616,795 bytes from stream 0
Thread 0 decompressed 65,282,515 bytes from stream 0
Taking decompressed data from thread 0
Reading ucomp header at 64
Fill_buffer stream 1 c_len 34,602,497 u_len 119,370,597 last_head 34,602,545
Starting thread 1 to decompress 34,602,497 bytes from stream 1
Reading ucomp header at 34,602,577
Fill_buffer stream 1 c_len 33,415,939 u_len 119,370,597 last_head 68,018,500
Starting thread 2 to decompress 33,415,939 bytes from stream 1
Reading ucomp header at 68,018,532
Fill_buffer stream 1 c_len 32,437,018 u_len 119,370,597 last_head 100,455,534
Starting thread 3 to decompress 32,437,018 bytes from stream 1
Reading ucomp header at 100,455,566
Fill_buffer stream 1 c_len 31,930,015 u_len 119,370,597 last_head 132,385,565
Starting thread 4 to decompress 31,930,015 bytes from stream 1
Reading ucomp header at 132,385,597
Fill_buffer stream 1 c_len 26,009,982 u_len 119,370,597 last_head 158,395,563
Starting thread 5 to decompress 26,009,982 bytes from stream 1
Reading ucomp header at 158,395,595
Fill_buffer stream 1 c_len 26,018,566 u_len 119,370,597 last_head 184,414,145
Starting thread 6 to decompress 26,018,566 bytes from stream 1
Reading ucomp header at 184,414,177
Fill_buffer stream 1 c_len 27,543,323 u_len 119,370,597 last_head 211,957,484
Starting thread 7 to decompress 27,543,323 bytes from stream 1
Reading ucomp header at 211,957,516
Fill_buffer stream 1 c_len 28,312,802 u_len 119,370,597 last_head 240,270,302
Starting thread 8 to decompress 28,312,802 bytes from stream 1
Reading ucomp header at 240,270,334
Fill_buffer stream 1 c_len 26,588,707 u_len 119,370,597 last_head 266,859,025
Starting thread 9 to decompress 26,588,707 bytes from stream 1
Reading ucomp header at 266,859,057
Fill_buffer stream 1 c_len 27,867,083 u_len 119,370,597 last_head 294,726,124
Starting thread 10 to decompress 27,867,083 bytes from stream 1
Reading ucomp header at 294,726,156
Fill_buffer stream 1 c_len 26,201,782 u_len 119,370,597 last_head 320,927,922
Starting thread 11 to decompress 26,201,782 bytes from stream 1
Thread 6 decompressed 119,370,597 bytes from stream 1
Thread 5 decompressed 119,370,597 bytes from stream 1
Thread 9 decompressed 119,370,597 bytes from stream 1
Thread 11 decompressed 119,370,597 bytes from stream 1
Thread 8 decompressed 119,370,597 bytes from stream 1
Thread 10 decompressed 119,370,597 bytes from stream 1
Thread 7 decompressed 119,370,597 bytes from stream 1
Thread 3 decompressed 119,370,597 bytes from stream 1
Thread 2 decompressed 119,370,597 bytes from stream 1
Thread 4 decompressed 119,370,597 bytes from stream 1
Thread 1 decompressed 119,370,597 bytes from stream 1
Taking decompressed data from thread 1
  0%       0.00 /     19.63 GB
Reading ucomp header at 320,927,954
Fill_buffer stream 1 c_len 25,591,468 u_len 119,370,597 last_head 346,519,406
Starting thread 1 to decompress 25,591,468 bytes from stream 1
Taking decompressed data from thread 2
Reading ucomp header at 346,519,438
Fill_buffer stream 1 c_len 28,612,218 u_len 119,370,597 last_head 375,131,640
Starting thread 2 to decompress 28,612,218 bytes from stream 1
Taking decompressed data from thread 3
Reading ucomp header at 375,131,672
Fill_buffer stream 1 c_len 26,820,374 u_len 119,370,597 last_head 401,952,030
Starting thread 3 to decompress 26,820,374 bytes from stream 1
Taking decompressed data from thread 4
Reading ucomp header at 401,952,062
Fill_buffer stream 1 c_len 27,762,985 u_len 119,370,597 last_head 429,715,031
Starting thread 4 to decompress 27,762,985 bytes from stream 1
Taking decompressed data from thread 5
Reading ucomp header at 429,715,063
Fill_buffer stream 1 c_len 24,834,895 u_len 119,370,597 last_head 454,549,942
Starting thread 5 to decompress 24,834,895 bytes from stream 1
Taking decompressed data from thread 6
Reading ucomp header at 454,549,974
Fill_buffer stream 1 c_len 28,099,866 u_len 119,370,597 last_head 482,649,824
Starting thread 6 to decompress 28,099,866 bytes from stream 1
Taking decompressed data from thread 7
Reading ucomp header at 482,649,856
Fill_buffer stream 1 c_len 27,600,791 u_len 119,370,597 last_head 510,250,631
Starting thread 7 to decompress 27,600,791 bytes from stream 1
Taking decompressed data from thread 8
Reading ucomp header at 510,250,663
Fill_buffer stream 1 c_len 25,016,315 u_len 119,370,597 last_head 535,266,962
Starting thread 8 to decompress 25,016,315 bytes from stream 1
Taking decompressed data from thread 9
Reading ucomp header at 535,266,994
Fill_buffer stream 1 c_len 27,745,130 u_len 119,370,597 last_head 563,012,108
Starting thread 9 to decompress 27,745,130 bytes from stream 1
Taking decompressed data from thread 10
Thread 1 decompressed 119,370,597 bytes from stream 1
Reading ucomp header at 563,012,140
Fill_buffer stream 1 c_len 27,421,858 u_len 119,370,597 last_head 590,433,982
Starting thread 10 to decompress 27,421,858 bytes from stream 1
Taking decompressed data from thread 11
Reading ucomp header at 590,434,014
Fill_buffer stream 1 c_len 27,384,681 u_len 119,370,597 last_head 617,818,679
Starting thread 11 to decompress 27,384,681 bytes from stream 1
Taking decompressed data from thread 1
Thread 2 decompressed 119,370,597 bytes from stream 1
Reading ucomp header at 617,818,711
Fill_buffer stream 1 c_len 26,450,256 u_len 119,370,597 last_head 644,268,951
Starting thread 1 to decompress 26,450,256 bytes from stream 1
Taking decompressed data from thread 2
Thread 3 decompressed 119,370,597 bytes from stream 1
Reading ucomp header at 644,268,983
Fill_buffer stream 1 c_len 26,875,982 u_len 119,370,597 last_head 671,144,949
Starting thread 2 to decompress 26,875,982 bytes from stream 1
Taking decompressed data from thread 3
Thread 5 decompressed 119,370,597 bytes from stream 1
Thread 4 decompressed 119,370,597 bytes from stream 1
Reading ucomp header at 671,144,981
Fill_buffer stream 1 c_len 27,446,030 u_len 119,370,597 last_head 698,590,995
Starting thread 3 to decompress 27,446,030 bytes from stream 1
Taking decompressed data from thread 4
Reading ucomp header at 698,591,027
Fill_buffer stream 1 c_len 25,853,843 u_len 119,370,597 last_head 724,444,854
Starting thread 4 to decompress 25,853,843 bytes from stream 1
Taking decompressed data from thread 5
Thread 6 decompressed 119,370,597 bytes from stream 1
Thread 7 decompressed 119,370,597 bytes from stream 1
Reading ucomp header at 724,444,886
Fill_buffer stream 1 c_len 25,505,909 u_len 119,370,597 last_head 749,950,779
Starting thread 5 to decompress 25,505,909 bytes from stream 1
Taking decompressed data from thread 6
Thread 8 decompressed 119,370,597 bytes from stream 1
 15%       3.09 /     19.63 GB
Reading ucomp header at 749,950,811
Fill_buffer stream 1 c_len 29,476,135 u_len 119,370,597 last_head 779,426,930
Starting thread 6 to decompress 29,476,135 bytes from stream 1
Taking decompressed data from thread 7
Thread 9 decompressed 119,370,597 bytes from stream 1
Reading ucomp header at 779,426,962
Fill_buffer stream 1 c_len 27,309,061 u_len 119,370,597 last_head 806,736,007
Starting thread 7 to decompress 27,309,061 bytes from stream 1
Taking decompressed data from thread 8
Thread 10 decompressed 119,370,597 bytes from stream 1
Thread 11 decompressed 119,370,597 bytes from stream 1
Reading ucomp header at 806,736,039
Fill_buffer stream 1 c_len 26,003,083 u_len 119,370,597 last_head 832,739,106
Starting thread 8 to decompress 26,003,083 bytes from stream 1
Taking decompressed data from thread 9
Thread 1 decompressed 119,370,597 bytes from stream 1
Reading ucomp header at 832,739,138
Fill_buffer stream 1 c_len 26,106,615 u_len 119,370,597 last_head 858,845,737
Starting thread 9 to decompress 26,106,615 bytes from stream 1
Taking decompressed data from thread 10
Thread 2 decompressed 119,370,597 bytes from stream 1
Reading ucomp header at 858,845,769
Fill_buffer stream 1 c_len 27,535,661 u_len 119,370,597 last_head 886,381,414
Starting thread 10 to decompress 27,535,661 bytes from stream 1
Taking decompressed data from thread 11
Thread 3 decompressed 119,370,597 bytes from stream 1
Thread 4 decompressed 119,370,597 bytes from stream 1
Reading ucomp header at 886,381,446
Fill_buffer stream 1 c_len 27,819,602 u_len 119,370,597 last_head 914,201,032
Starting thread 11 to decompress 27,819,602 bytes from stream 1
Taking decompressed data from thread 1
Thread 5 decompressed 119,370,597 bytes from stream 1
Reading ucomp header at 914,201,064
Fill_buffer stream 1 c_len 24,693,511 u_len 119,370,597 last_head 938,894,559
Starting thread 1 to decompress 24,693,511 bytes from stream 1
Taking decompressed data from thread 2
Reading ucomp header at 938,894,591
Fill_buffer stream 1 c_len 29,300,303 u_len 119,370,597 last_head 968,194,878
Starting thread 2 to decompress 29,300,303 bytes from stream 1
Taking decompressed data from thread 3
Thread 6 decompressed 119,370,597 bytes from stream 1
Reading ucomp header at 968,194,910
Fill_buffer stream 1 c_len 28,444,764 u_len 119,370,597 last_head 996,639,658
Starting thread 3 to decompress 28,444,764 bytes from stream 1
Taking decompressed data from thread 4
Thread 7 decompressed 119,370,597 bytes from stream 1
Thread 8 decompressed 119,370,597 bytes from stream 1
Reading ucomp header at 996,639,690
Fill_buffer stream 1 c_len 26,420,722 u_len 119,370,597 last_head 1,023,060,396
Starting thread 4 to decompress 26,420,722 bytes from stream 1
Taking decompressed data from thread 5
Thread 9 decompressed 119,370,597 bytes from stream 1
Reading ucomp header at 1,023,060,428
Fill_buffer stream 1 c_len 27,378,184 u_len 119,370,597 last_head 1,050,438,596
Starting thread 5 to decompress 27,378,184 bytes from stream 1
Taking decompressed data from thread 6
Reading ucomp header at 1,050,438,628
Fill_buffer stream 1 c_len 25,351,761 u_len 119,370,597 last_head 1,075,790,373
Starting thread 6 to decompress 25,351,761 bytes from stream 1
Taking decompressed data from thread 7
Thread 10 decompressed 119,370,597 bytes from stream 1
Unable to decompress entirely in ram, will use physical files
Failed to write 5,726,589,567 bytes to fd_out in write_fdout
Deleting broken file /Users/nick/Downloads/kc.tar
Fatal error - exiting

This is consistent what's stated in macOS's write(2) manpage:

write() and pwrite() will fail if the parameter nbyte exceeds INT_MAX, and they do not attempt a partial write.

5,726,589,567 bytes is larger than 2,147,483,647 bytes

from lrzip-next.

pete4abw avatar pete4abw commented on June 5, 2024

Thank you, Nick. I am not sure it's a MAC thing though. write is returning an error. This could be a number of things. I'll check here and see. I have some really big files too. I'll wash it through the debugger, check errno. Make sure your temp directory has space enough, or set tmpdir.

from lrzip-next.

asdfugil avatar asdfugil commented on June 5, 2024

the errno appears to be 22, which is Invalid argument on macOS.

Unable to decompress entirely in ram, will use physical files
Process 12447 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.2
    frame #0: 0x0000000100009590 lrzip-next`fatal(control=0x0000000100085d50, line=685, file="lrzip.c", func="write_fdout", format="Failed to write %'lld bytes to fd_out in write_fdout\n") at util.h:38:2
   35  	{
   36  		va_list ap;
   37  		/* lrzip library callback code removed */
-> 38  		va_start(ap, format);
   39  		vfprintf(stderr, format, ap);
   40  		va_end(ap);
   41  		fatal_exit((rzip_control*)control);
Target 0: (lrzip-next) stopped.
(lldb) po *(int*)__error()
22
(lldb) po (char*)strerror(22)
"Invalid argument"

from macOS intro(2):

22 EINVAL Invalid argument. Some invalid argument was supplied. (For example, specifying an undefined signal to a signal or kill function).

from lrzip-next.

pete4abw avatar pete4abw commented on June 5, 2024

Interesting. But that's not part of the write function. That's the fatal macro that prints an error. I appreciate the work. I'll take a look sometime this week. Thank you again.

from lrzip-next.

asdfugil avatar asdfugil commented on June 5, 2024

since the breakpoint is at the start of the fatal function, and fatal is called immediately after the failed write, errno is not overwritten yet at that time

from lrzip-next.

pete4abw avatar pete4abw commented on June 5, 2024

I am unable to duplicate on x86_64.

Unable to decompress entirely in ram, will use physical files
Thread 7 decompressed 306,266,112 bytes from stream 1
nmemb in write_fdout = 5,512,696,498. Return value = 2,147,479,552
nmemb in write_fdout = 3,365,216,946. Return value = 2,147,479,552
nmemb in write_fdout = 1,217,737,394. Return value = 1,217,737,394

As you can see, on this system, write performs properly, limiting writing to 2G and returning properly bytes written. So I need to understand better about MAC. I'm guessing that write on MAC is different. lrzip-next does not support 32 bit at all, but maybe your write function does not accept large values > 2G?

Can you compile and run this code?

#include <stdio.h>
#include <limits.h>

int main(int argc,char **argv)
{
        printf("Size_t = %d\n", sizeof(size_t));
        printf("SSize_t = %d\n", sizeof(ssize_t));
        printf("SIZE_MAX = %llu\n", ULONG_MAX);
        printf("SSIZE_MAX = %lld\n", LONG_MAX);
}

On my system, I see this:

$ ./a.out
Size_t = 8
SSize_t = 8
SIZE_MAX = 18446744073709551615
SSIZE_MAX = 9223372036854775807

Depending on your output, we could put an #ifdef __APPLE__ and call the write_1g function. I just need to understand more about MAC write(). Thank you.

from lrzip-next.

asdfugil avatar asdfugil commented on June 5, 2024

macOS 14.2.1 (arm64)

Size_t = 8
SSize_t = 8
SIZE_MAX = 18446744073709551615
SSIZE_MAX = 922337203685477580

from lrzip-next.

pete4abw avatar pete4abw commented on June 5, 2024

macOS 14.2.1 (arm64)

Size_t = 8
SSize_t = 8
SIZE_MAX = 18446744073709551615
SSIZE_MAX = 922337203685477580

Well, I hate kludges. And i cannot explain why the arm64 write() function does not perform as expected. But at least we confirmed a bug. One final thing. Can you show the headers of libc?

On my system readelf -h [libdir]/libc.so.6

ELF Header:
Magic: 7f 45 4c 46 02 01 01 03 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - GNU
ABI Version: 0
Type: DYN (Shared object file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x27390
Start of program headers: 64 (bytes into file)
Start of section headers: 1918040 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 14
Size of section headers: 64 (bytes)
Number of section headers: 64
Section header string table index: 63

from lrzip-next.

asdfugil avatar asdfugil commented on June 5, 2024

The libc and all other system libraries and frameworks on macOS are all linked together in files that are called “dyld shared cache”. It’s not a cache despite it’s name.

from lrzip-next.

pete4abw avatar pete4abw commented on June 5, 2024

The libc and all other system libraries and frameworks on macOS are all linked together in files that are called “dyld shared cache”. It’s not a cache despite it’s name.

I'll see what I can do. I can pretty much guarantee an #ifdef __APPLE__ kludge. Maybe limit nmemb, or maybe call write_1g.

from lrzip-next.

asdfugil avatar asdfugil commented on June 5, 2024

github actions macOS runs on macOS x86_64, , write() there doesn't work for size bigger than INT_MAX either

https://github.com/asdfugil/macos-write-test/actions/runs/7655319378/job/20861107940

from lrzip-next.

pete4abw avatar pete4abw commented on June 5, 2024

I'll plug in your code, but limit it to APPLE so it won't impact other systems where this works properly.

from lrzip-next.

pete4abw avatar pete4abw commented on June 5, 2024

Fix proposed in branch whats-next. Please review and comment.

from lrzip-next.

asdfugil avatar asdfugil commented on June 5, 2024

doesn't work because the correct macro is __APPLE__, not APPLE__

from lrzip-next.

pete4abw avatar pete4abw commented on June 5, 2024

Because I am an idiot. I manually changed the APPLE macro so I could test, and just forgot to undo it. numb, I have no idea what I was thinking. Thank you for taking a look. Pushing update.

diff --git a/src/lrzip.c b/src/lrzip.c
index eefc4d9..fcdbb82 100644
--- a/src/lrzip.c
+++ b/src/lrzip.c
@@ -678,9 +678,9 @@ bool write_fdout(rzip_control *control, void *buf, i64 len)
 
        while (len > 0) {
                nmemb = len;
-# ifdef APPLE__
+# ifdef __APPLE__
                if (nmemb > 0x7FFF0000)
-                      numb = 0x7FFF0000;
+                      nmemb = 0x7FFF0000;
 # endif
                ret = write(control->fd_out, offset_buf, (size_t)nmemb);
                /* error if ret == -1 only. Otherwise, buffer not wholly written */

from lrzip-next.

asdfugil avatar asdfugil commented on June 5, 2024

I can confirm that it all works on mac now

from lrzip-next.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.