mre / fcat Goto Github PK

View Code? Open in Web Editor NEW

264.0 10.0 8.0 458 KB

A 3x faster implementation of cat, using splice

Home Page: https://endler.dev/2018/fastcat

License: Apache License 2.0

Dockerfile 11.05% Makefile 4.04% Shell 7.31% Rust 77.60%

splice linux android cat unix

fcat's Introduction

😼 fcat

fcat, short for fastcat, is a cat implementation in Rust using Linux's splice syscall.
With that little trick, it's more than three times as fast as the system cat in our benchmarks.
Read the announcement here.

⚠️ This project is currently broken on newer Linux versions (5.9+) because of some changes
concerning the splice system call. (See here and here.) This can't be fixed unless changes to the kernel get made.

Performance

cat myfile | pv -r > /dev/null
[1.90GiB/s]

fcat myfile | pv -r > /dev/null
[5.90GiB/s]

Installation

Note: Only works on Linux.
(But you can send me a pull request for other operating systems.)

cargo install fcat

Usage

fcat file1 file2 file3

Project goals

Be the fastest cat in town.
Be a drop-in replacement for (POSIX) cat.

Non-goals

Provide any additional functionality other than what cat provides.
If you're looking for a more beautiful cat, check out bat.

Known issues

If you run fcat /dev/zero >> myfile, it will fail with exit code EINVAL because, according to the splice manpage: "The target file is opened in append mode."

Trivia

You probably won't ever need this, but it's a fun little experiment.
Still, I wonder why this is not part of e.g. GNU cat...
What I like the most about the project is the logo.

License

fcat is licensed under either of

Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

fcat's People

Contributors

Stargazers

Watchers

Forkers

vbrandl dzamlo arnidagur baitcenter zhdd1124 iq-scm dmgolembiowski skull-squadron

fcat's Issues

Improve perfomance even more

Suggestions here: https://www.reddit.com/r/rust/comments/93fbrj/fascat_a_faster_cat_implementation_using_splice/e3dl0f3/

Use only one splice if stdout is already a pipe

stdout is often already a pipe. In this case it's possible to directly splice to it, without an intermediate pipe.

It may be interesting to compare that with the current implementation. I wouldn't expect much difference, but it may be interesting nonetheless.

travis support

Now that we have some tests (thanks @vbrandl!), we should also set up a proper travis environment that will run the tests after every change.

Thank you

Thank you for open-sourcing this.

Also, the logo is indeed really awesome!

Fix issues with assert_cmd crate

The version bump in #18 failed because assert_cmd changed their api.
We should fix that by updating the code that depends on it.

Here's the output of cargo test triggered by Github Actions:

error[E0599]: no method named `set_cmd` found for type `assert_cmd::assert::Assert` in the current scope
   --> src/main.rs:105:64
    |
105 |             let cmd = Command::main_binary().unwrap().assert().set_cmd(path);
    |                                                                ^^^^^^^

error[E0599]: no method named `set_cmd` found for type `assert_cmd::assert::Assert` in the current scope
   --> src/main.rs:118:18
    |
118 |                 .set_cmd(format!("{} {}", path0, path1));
    |                  ^^^^^^^

error[E0599]: no method named `set_stdin` found for type `assert_cmd::assert::Assert` in the current scope
   --> src/main.rs:130:18
    |
130 |                 .set_stdin(content.clone());
    |                  ^^^^^^^^^

Fix issues with proptest crate

We have issues like that when bumping it to the latest version (see #22):

error[E0277]: the trait bound `rand_xorshift::XorShiftRng: rand_core::SeedableRng` is not satisfied
   --> /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/rand-0.6.5/src/deprecated.rs:240:5
    |
240 |     type Seed = <::rand_xorshift::XorShiftRng as SeedableRng>::Seed;
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `rand_core::SeedableRng` is not implemented for `rand_xorshift::XorShiftRng`

error[E0277]: the trait bound `rand_xorshift::XorShiftRng: rand_core::SeedableRng` is not satisfied

fcat can't print anything in my system

Hi, maybe it's some dependency that dependabot broke but my fcat is not working anymore, it gives me this error:

Thanks in advance for the support!

EINVAL error when outputing to a file in append mode.

If you run fcat /dev/zero >> /tmp/myfile, it fail with EINVAL, because, according to the splice manpage: "The target file is opened in append mode."

It thinks should be noted somewhere that this is a limitation

thread 'main' panicked at 'called `Result::unwrap()`

After installation, I'm getting the following error:

❯ rustc -V
rustc 1.30.0-nightly (73c78734b 2018-08-05)

❯ RUST_BACKTRACE=1 fcat ~/.zshrc
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Sys(EINVAL)', libcore/result.rs:945:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:71
             at libstd/sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:211
   3: std::panicking::default_hook
             at libstd/panicking.rs:227
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:463
   5: std::panicking::begin_panic_fmt
             at libstd/panicking.rs:350
   6: rust_begin_unwind
             at libstd/panicking.rs:328
   7: core::panicking::panic_fmt
             at libcore/panicking.rs:71
   8: core::result::unwrap_failed
   9: fcat::main
  10: std::rt::lang_start::{{closure}}
  11: std::panicking::try::do_call
             at libstd/rt.rs:59
             at libstd/panicking.rs:310
  12: __rust_maybe_catch_panic
             at libpanic_unwind/lib.rs:105
  13: std::rt::lang_start_internal
             at libstd/panicking.rs:289
             at libstd/panic.rs:374
             at libstd/rt.rs:58
  14: main
  15: __libc_start_main
  16: _start

Compilation error on Apple Silicon Mac

>> cargo install fcat --verbos
.
.
.
     Running `rustc --crate-name fcat /Users/reportaman/.cargo/registry/src/github.com-1ecc6299db9ec823/fcat-0.1.0/src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C embed-bitcode=no -C metadata=170953d2de8e5e5f -C extra-filename=-170953d2de8e5e5f --out-dir /var/folders/40/g1hxt47j5sn3wg34689bfpnr0000gn/T/cargo-installhV87a2/release/deps -L dependency=/var/folders/40/g1hxt47j5sn3wg34689bfpnr0000gn/T/cargo-installhV87a2/release/deps --extern nix=/var/folders/40/g1hxt47j5sn3wg34689bfpnr0000gn/T/cargo-installhV87a2/release/deps/libnix-e97ee1ac4be0467d.rlib --cap-lints allow`
error[E0432]: unresolved imports `nix::fcntl::splice`, `nix::fcntl::SpliceFFlags`
 --> /Users/reportaman/.cargo/registry/src/github.com-1ecc6299db9ec823/fcat-0.1.0/src/main.rs:8:18
  |
8 | use nix::fcntl::{splice, SpliceFFlags};
  |                  ^^^^^^  ^^^^^^^^^^^^ no `SpliceFFlags` in `fcntl`
  |                  |
  |                  no `splice` in `fcntl`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0432`.
error: failed to compile `fcat v0.1.0`, intermediate artifacts can be found at `/var/folders/40/g1hxt47j5sn3wg34689bfpnr0000gn/T/cargo-installhV87a2`

Caused by:
  could not compile `fcat`

System info:

>> neofetch
                    'c.          [email protected] 
                 ,xNMM.          ------------------------------ 
               .OMMMMo           OS: macOS 11.2 20D64 arm64 
               OMMM0,            Host: Macmini9,1 
     .;loddo:' loolloddol;.      Kernel: 20.3.0 
   cKMMMMMMMMMMNWMMMMMMMMMM0:     
 .KMMMMMMMMMMMMMMMMMMMMMMMWd.     
 XMMMMMMMMMMMMMMMMMMMMMMMX.      Shell: zsh 5.8 
;MMMMMMMMMMMMMMMMMMMMMMMM:        
:MMMMMMMMMMMMMMMMMMMMMMMM:       DE: Aqua 
.MMMMMMMMMMMMMMMMMMMMMMMMX.      WM: Quartz Compositor 
 kMMMMMMMMMMMMMMMMMMMMMMMMWd.    WM Theme: Blue (Dark) 
 .XMMMMMMMMMMMMMMMMMMMMMMMMMMk   Terminal: Apple_Terminal 
  .XMMMMMMMMMMMMMMMMMMMMMMMMK.   
    kMMMMMMMMMMMMMMMMMMMMMMd     CPU: Apple M1 
     ;KMMMMMMMWXXWMMMMMMMk.      GPU: Apple M1 
       .cooc,.    .,coo:.

`EMFILE` error when `cat`int too many files because pipes are not closed.

For each file, there is a call to the pipe function to create a pipe. But the resulting pipe are never closed. This means than if you try to cat too man files, one of the call to pipe will fail with an EMFILE error (and the unwrap will panic).

The simple obvious solution is to just close the pipe each times.

But a more efficient solution would be to reuse the pipe. This would reduce the number of syscall per file by 3 (the pipe and the two close), which could have an impact if you try to cat a lot of small files.