ouch currently determines the compression format of its input files through their extensions, which isn't ideal

Currently, we only test archive formats with multiple regular random files, there's no nested folders testing, so for that formats the tests need to be refactored.

This is important before a major release. Just to make sure that the errors are being correctly detected, from the CLI all the way to the conflict in output files, or folders.

The API for commands::run is already correctly exposed for testing, here's an example:

https://github.com/vrmiguel/ouch/blob/master/tests/compress_and_decompress.rs#L93-L97

It's a subtle change, but:

After the [Y/n] part, the n written should have some color to emphasize it, like magenta.

#![warn(missing_docs)] at the lib crate root will force us to document the codebase to enable other people to help us developing ouch in the future.

Not motivated to complete this one, but we spent too much time on the actual code, and not even 5% of that with documenting.

Not prudent, ARRGHHH. Why. AI needs to figure this out for the humanity, Google has been listening to me coding all day long, so maybe it does understand all our codebase? Hey alexa, hey alexa

Relevant branch

• ouch file.tar.gz -o new-folder

  • Expected: create new-folder if it doesn't exist and decompress the files in there
  • What currently happens: ouch ignores the flag and saves to pwd

• ouch file.tar.gz --output new-folder

  • Expected: ouch src.tar.lzma --output new-src
  • What currently happens: [ERROR] cannot compress to 'new-src', likely because it has an unsupported (or missing) extension.

When checking the CI for specific commit (click on the ✔️ and then details):

The MacOS artifact is not there:

Even if it succeeded.

There is currently no way to compress an file.tar into file.tar.gz by only adding the .gz part, ouch will detect two extensions, tar and gz, repeating the .tar process for that file that was already in that format.

So basically, ouch compress file.tar.gz file.tar.gz.xz generates a file of format .tar.gz.tar.gz.xz, instead of .tar.gz.xz.

Steps to reproduce:

1. wget -O revealjs.tar.gz https://github.com/hakimel/reveal.js/archive/3.9.2.tar.gz
2. Try to uncompres it

❯ ouch ./revealjs.tar.gz
[INFO] "/home/sibi/github/transcripts/justl/revealjs.tar.gz" extracted into memory (4.25 MB).
[INFO] attempting to decompress "revealjs.tar"
[INFO] "./pax_global_header" extracted. (52.00 B)
thread 'main' panicked at 'No such file or directory (os error 2)', src/error.rs:96:45
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Version used:

❯ ouch --version
ouch 0.1.5

Originally posted by @vrmiguel in #89 (comment)

Problem

The flag --output is not working properly when compressing one file into one non-archive format compressed file.

# Works
$ cargo run -- c src/main.rs main.rs.xz 

# Panics
$ cargo run -- d main.rs.xz -o some-new-folder
thread 'main' panicked at 'No such file or directory (os error 2)', src/error.rs:173:45
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

We have a custom argparsing library called oof, it's a subcrate in ouch's workspace, so we have to certify ourselves that argparsing is being done correctly.

Would be nice to have. Especially if it defaulted to multithreaded operation (unlike upstream).

Basically as an alias of .tar.gz.

As asked in #5 (here).

Will be supported after #28 is fixed.

Problem

Currently, it's not possible to decompress a file partially.

Take archive.tar.gz.xz as an example, you might want to decompress it into:

• archive.tar.gz, without the .xz part, or
• archive.tar, without the .gz.xz part.

Current workaround

Decompress and compress it again.

ouch decompress archive.tar.gz.xz -o result
cd result
ouch compress * .* archive.tar 

Note that the shell expansion .* is necessary in case you have any hidden files.

Suggested new solution

Adding a --partial (-p) flag that takes the extensions as a parameter.

# Only the .xz part
ouch decompress archive.tar.gz.xz --partial xz

# The file created:
ls archive.tar.gz

Another example:

# Correctly decompressing .xz.gz
ouch decompress file.txt.lzma.bz.xz.gz --partial xz.gz

# The file created:
ls file.txt.lzma.bz

First, check if the suffix is really present in the file passed as argument, if multiple files given, then all files should have the suffix, otherwise, give an usage error message.

Some formats, like .xz can only compress one single file, as opposed to .zip that compresses multiple and bundle them together in one file.

Currently we are only testing formats that create archives, and throwing single file on top of it.

Maybe we should test the single file formats separately.

.tar.xz already indirectly tests for .xz, that is enough for compression and decompression purposes, but we want to make sure that the CLI won't panic in any specific case.

So aur helpers users can install it like:

yay -S ouch
# while also being possible:
# yay -S ouch-bin

Using the suffix -bin is a convention for when the installation just downloads a binary.

NOTE: passing ouch as an argument instead of the full name is achievable by using the provides field in the PKGBUILD file.

NOTE²: maybe in the future we will ship it with shell completions too.

Each color could have it's function created with a macro, the function would receive the color &str as the argument and use conditional compilation inside in an cfg!() if block, which is resolved and optimized out in compile time.

After posting ouch on reddit, we received a lot of feedback.

People found the usage to not be obviously trivial, because we are not using two subcommands.

Suggestion:

ouch compress ...
ouch decompress ...

And the aliases:

ouch c ...
ouch d ...

It will behave much like cargo it does not have a "default" command like ouch do now, and everybody likes it's usage.

Instead of panicking like this:

# src/ is a folder
$ cargo run -q -- c src src.xz
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/commands.rs:70:85
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Should give a helpful error message, saying that its impossible to compress a folder with .xz.

Same applies for .gz, .bz and .zst.

Running ouch c src/ t.bz currently yields in

thread 'main' panicked at 'internal error: entered unreachable code', src/commands.rs:165:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

We once had an error message for this. We should bring it back.

• High priority.
• Medium priority.
• Low Priority.

When decompress used to be the default command, running ouch would prompt up the help menu, as it also does now, however, since we made the decompress command explicit, now it needs a default behaviour to throw an error message

Right now it panics:

$ ouch decompress
thread 'main' panicked at 'not yet implemented: Complain that no decompression arguments were given.', src/cli.rs:128:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

These targets were previously in our CI but deactivated in #82, because of problems building Zstd (with the thin feature
) for these targets.

thin needs to be disabled those two targets to make this two CI targets work again.

That will give more context to errors when dealing with filesystem operations.

To replace it:

// Replace this
use std::fs;

// By this
use fs_err as fs;

Hopefully adding checks too to disable .zip.zip, or any multiple file archive formats in a wrong way.

We should only expect .zip and/or .tar appearing once in the extensions? @vrmiguel

I think that would be a nice idea, .tar.tar and .tar.gz.zip does not seems useful, and we fell to the problem of not having in-memory decompression.

Planned flags:

--level LEVEL_NUMBER

From 1 to 10, ouch should "translate" that range for each format, considering that some formats use higher numbers than others.

--fast

Less intensive compressing, we can seek for the best defaults aiming for 50% of time taken (run in half the time).

--slow

More intensive compressing, takes longer, we shall look for defaults that take up to double the time, but also, we'll need to establish a RAM limit.

Some compression methods have a memory usage table that explains how much a certain level can use.

He are repeating ourselves in the Display implementation of crate::Error.

@vrmiguel Could we have a UserError (or FinalUserError) that uses the builder pattern and formats everything for us?

Something like:

FinalError::with_reason("Could not ride the tank")
    .add_line("Because you don't have a license")
    .add_line("Since you lost your ferrari")
    .hint("Try again with `sudo`.")
    .crash()

Would display a colorful message:

[ERROR]: Could not ride the tank

• Because you don't have a license.
• Maybe try again, without crashing a ferrari.

🌹 Hint 🌹 🚀 💯
Try again with sudo.

This percentage will vary depending on the terminal emulator you are using, I tried it with alacritty, which is GPU accelerated, and it also slowed it down by the same quantities (in some scenarios).

Differences were calculated by decompressing and compressing the linux arch package with and without redirecting the output to /dev/null with "&> /dev/null" at the end of the command.

This probably means that that we are flushing the output too many times, using a bufwriter will probably solve it.

However, there are some problems with regards to it, we need to make sure that stderr, stdout and stdin are always in sync. If we fail to sync stdin, we will wait for user input but the user won't see the message asking for it.

I'm actually not sure how difficult it is to solve this.

Hello, I was giving #54 a try when I stumbled on to this.

Running ouch c file file.gz for example panics because it hits an unreachable!():
thread 'main' panicked at 'internal error: entered unreachable code', src/commands.rs:176:18

Is this correct behavior?

Suggestions:

• Should change word "bundle" to "archive".
• When the destination is '.', output current directory instead.
• Remove the './' at the beginning of each file, they are pointless: './file'. .
• The command is called decompress, but here we have "uncompress", seems to be a bit off.

The pre-compiled macOS binary appears to be x86 only. Would it be possible to have a universal build supporting ARM architecture?
When building from source using cargo install ouch rustup first installs an x86-64 toolchain (in addition to the arm64 toolchain) on my computer and later fails during the compilation:

   Compiling ouch v0.1.5
error: linking with `cc` failed: exit status: 1
  |
  = note: "cc" "-m64" "-arch" "x86_64" "-L" "~/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "/var/folders/_g/nfj48t2948s35nnrcs4p3q_c0000gn/T/cargo-install1namSU/release/deps/ouch-c1fc9f0cbd401a2e.ouch.1mb7mks5-cgu.0.rcgu.o" "-o" "/var/folders/_g/nfj48t2948s35nnrcs4p3q_c0000gn/T/cargo-install1namSU/release/deps/ouch-c1fc9f0cbd401a2e" "-Wl,-dead_strip" "-nodefaultlibs" "-L" "/var/folders/_g/nfj48t2948s35nnrcs4p3q_c0000gn/T/cargo-install1namSU/release/deps" "-L" "/var/folders/_g/nfj48t2948s35nnrcs4p3q_c0000gn/T/cargo-install1namSU/release/build/bzip2-sys-e23d54bac5dc72d9/out/lib" "-L" "/opt/homebrew/Cellar/xz/5.2.5/lib" "-L" "/Users/salomon/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "/var/folders/_g/nfj48t2948s35nnrcs4p3q_c0000gn/T/rustcnCaz4b/libbzip2_sys-ad219373a4b8ba7e.rlib" "/Users/salomon/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-160a7dd17f2dc334.rlib" "-llzma" "-liconv" "-lSystem" "-lresolv" "-lc" "-lm" "-liconv"
  = note: ld: warning: ignoring file /opt/homebrew/Cellar/xz/5.2.5/lib/liblzma.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
          Undefined symbols for architecture x86_64:
            "_lzma_code", referenced from:
                _$LT$xz2..read..XzDecoder$LT$R$GT$$u20$as$u20$std..io..Read$GT$::read::hb1428dcfb9c41431 in ouch-c1fc9f0cbd401a2e.ouch.1mb7mks5-cgu.0.rcgu.o
                ouch::compressors::lzma::LzmaCompressor::compress_bytes::h94f82ba83697a63a in ouch-c1fc9f0cbd401a2e.ouch.1mb7mks5-cgu.0.rcgu.o
                xz2::write::XzEncoder$LT$W$GT$::try_finish::hcd128a9a358b26ee in ouch-c1fc9f0cbd401a2e.ouch.1mb7mks5-cgu.0.rcgu.o
            "_lzma_easy_encoder", referenced from:
                ouch::compressors::lzma::LzmaCompressor::compress_bytes::h94f82ba83697a63a in ouch-c1fc9f0cbd401a2e.ouch.1mb7mks5-cgu.0.rcgu.o
            "_lzma_auto_decoder", referenced from:
                ouch::decompressors::to_memory::DecompressorToMemory::decompress::hd679a1dec8865489 in ouch-c1fc9f0cbd401a2e.ouch.1mb7mks5-cgu.0.rcgu.o
            "_lzma_end", referenced from:
                ouch::decompressors::to_memory::DecompressorToMemory::decompress::hd679a1dec8865489 in ouch-c1fc9f0cbd401a2e.ouch.1mb7mks5-cgu.0.rcgu.o
                core::ptr::drop_in_place$LT$xz2..read..XzDecoder$LT$alloc..boxed..Box$LT$dyn$u20$std..io..Read$u2b$core..marker..Send$GT$$GT$$GT$::hf7ec9572769d8944 in ouch-c1fc9f0cbd401a2e.ouch.1mb7mks5-cgu.0.rcgu.o
                core::ptr::drop_in_place$LT$xz2..stream..Stream$GT$::h6048c3516c7203ec in ouch-c1fc9f0cbd401a2e.ouch.1mb7mks5-cgu.0.rcgu.o
                ouch::compressors::lzma::LzmaCompressor::compress_bytes::h94f82ba83697a63a in ouch-c1fc9f0cbd401a2e.ouch.1mb7mks5-cgu.0.rcgu.o
                core::ptr::drop_in_place$LT$xz2..write..XzEncoder$LT$alloc..vec..Vec$LT$u8$GT$$GT$$GT$::h5164a69ef0e186df in ouch-c1fc9f0cbd401a2e.ouch.1mb7mks5-cgu.0.rcgu.o
                core::ptr::drop_in_place$LT$xz2..stream..Stream$GT$::h92dfb9102f4d6f54 in ouch-c1fc9f0cbd401a2e.ouch.1mb7mks5-cgu.0.rcgu.o
          ld: symbol(s) not found for architecture x86_64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

Originally posted by @dcariotti in #84 (comment)

Problem

After running this:

ouch c 1.txt 2.txt archive.lz.tar

Ouch shows an error message:

[ERROR] Cannot compress to 'archive.lz.tar'.
 - You are trying to compress multiple files.
 - The compression format '.lz' cannot receive multiple files.
 - The only supported formats that archive files into an archive are .tar and .zip.

hint: Try inserting '.tar' or '.zip' before '.lz'.
hint: From: archive.lz.tar
hint:  To : archive.tar.lz.tar

However, if you follow what the hints suggest and use archive.tar.lz.tar:

ouch c 1.txt 2.txt archive.tar.lz.tar

You will receive the following error:

 - Found the format '.tar' in an incorrect position.
 - '.tar' can only be used at the start of the file extension.

hint: If you wish to compress multiple files, start the extension with '.tar'.
hint: Otherwise, remove '.tar' from 'archive.tar.lz.tar'.

Our hints are not ready for this specific errors.

Maybe it probably should just have said "you might want to move .tar to the start".

There are actually a lot of ways we can rethink those error messages to fix this issue, I'm not currently sure what's the best one.

By using atty we can easily check if the STDOUT stream is pointing to another file descriptor.

Currently, redirecting ouch output to a file makes it unreadable in a lot of programs, as terminal escape sequences are only interpreted by terminals, and only by some tools (may work with cat, but not with less or bat).

Unresolved questions:

1. Should we add an --color flag to overwrite this behavior (and the NO_COLOR env var)?

Also, because we are already talking about colors.

2. We could also add --no-color flag to deactivate colors, maybe you are using a terminal that's not working properly with it?

We should generate completions at least for zsh and bash.

Status update:

• #105.
• Add completions to releases.
• Install correctly via the installation script.

This issue DOES NOT affect any release of ouch, just running the development tests, however, it can turn out to be a pain for us developers in the future.

Spoiler: I HAVE NO CLUE OF WHAT IS GOING ON, and I've been trying to understand this issue for some time.

How to make it break

At the time of writing, this is (almost) the tests of archive compressing and decompressing (click here to jump to the code):

#[test]
/// Tests each format that supports multiple files with random input.
fn test_each_format() {
    test_compressing_and_decompressing_archive("tar");
    test_compressing_and_decompressing_archive("tar.gz");
    test_compressing_and_decompressing_archive("tar.bz");
    test_compressing_and_decompressing_archive("tar.bz2");
    test_compressing_and_decompressing_archive("tar.xz");
    test_compressing_and_decompressing_archive("tar.lz");
    test_compressing_and_decompressing_archive("tar.lzma");
    test_compressing_and_decompressing_archive("zip");
    test_compressing_and_decompressing_archive("zip.gz");
    test_compressing_and_decompressing_archive("zip.bz");
    test_compressing_and_decompressing_archive("zip.bz2");
    test_compressing_and_decompressing_archive("zip.xz");
    test_compressing_and_decompressing_archive("zip.lz");
    test_compressing_and_decompressing_archive("zip.lzma");
}

It works, but let's say I try to split some of these into different test functions.

#[test]
fn test_zip() {
    test_compressing_and_decompressing_archive("zip");
    test_compressing_and_decompressing_archive("zip.gz");
    test_compressing_and_decompressing_archive("zip.bz");
    test_compressing_and_decompressing_archive("zip.bz2");
    test_compressing_and_decompressing_archive("zip.xz");
    test_compressing_and_decompressing_archive("zip.lz");
    test_compressing_and_decompressing_archive("zip.lzma");
}

#[test]
fn test_tar() {
    test_compressing_and_decompressing_archive("tar");
    test_compressing_and_decompressing_archive("tar.gz");
    test_compressing_and_decompressing_archive("tar.bz");
    test_compressing_and_decompressing_archive("tar.bz2");
    test_compressing_and_decompressing_archive("tar.xz");
    test_compressing_and_decompressing_archive("tar.lz");
    test_compressing_and_decompressing_archive("tar.lzma");
}

Or even one test per test_compressing_and_decompressing_archive call.

What breaks?

When we run more than one test at once, a lot of stuff break, here are some symptoms:

• A lot of "File or directory not found"
  • Files passed for compression disappear before opening (including opening for .zip archive building)
  • Decompressed files may be disappearing too
• Compared file contents after compressing and decompressing suddenly mismatch

My suspections:

I suspect it is caused because different tests are run in different threads.

Ouch is 100% unsafe free, and I can't find where we could be causing this problem, so I do suspect it has something to do with some library we are using, even thought it is very unlikely.

• Maybe tempfile::TempDir is messing up with drop in different threads?
• rand::SmallRng is very unlikely to have an non-documented bug like this.
• Any compression crate, like flate2, bzip2, tar, xz2 or zip?

We currently have a lot of "todo"s left in the code that needs to be solved.