lotabout / rargs Goto Github PK

View Code? Open in Web Editor NEW

442.0 12.0 21.0 57 KB

xargs + awk with pattern matching support. `ls *.bak | rargs -p '(.*)\.bak' mv {0} {1}`

License: MIT License

Rust 100.00%

rargs's Introduction

Rargs is kind of xargs + awk with pattern-matching support.

Installation

Mac OS

brew install rargs

Nix

nix-env -i rargs

(Currently available in unstable channel)

Binary

Download in the Release Page and put it in your PATH after uncompress.

Using Cargo

cargo install --git https://github.com/lotabout/rargs.git

Example usage

Batch rename files

Suppose you have several backup files whose names match the pattern <scriptname>.sh.bak, and you want to map each filename back to <scriptname>.sh. We want to do it in a batch, so xargs is a natural choice, but how do we specify the name for each file? I believe there is no easy way.

With rargs, however, you are able to do:

ls *.bak | rargs -p '(.*)\.bak' mv {0} {1}

Here {0} refers to the whole input line, while {1} refers to the first group captured in the regular expression.

Batch download

I had a bunch of URLs and their corresponding target filenames stored in a CSV file:

URL1,filename1
URL2,filename2

I hoped there was a simple way to download and save each URL with its specified filename. With rargs there is:

cat download-list.csv | rargs -p '(?P<url>.*),(?P<filename>.*)' wget {url} -O {filename}

Here (?P<group_name>...) assigns the name group_name to the captured group. This can then be referred to as {group_name} in the command.

AWK replacement?

Suppose you have an xSV file with lots of columns, and you only want to extract and format some of them, e.g.:

nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false
root:*:0:0:System Administrator:/var/root:/bin/sh
daemon:*:1:1:System Services:/var/root:/usr/bin/false

Here's an example of how rargs can be used to process it:

$ cat /etc/passwd | rargs -d: echo -e 'id: "{1}"\t name: "{5}"\t rest: "{6..::}"'
id: "nobody"     name: "Unprivileged User"       rest: "/var/empty:/usr/bin/false"
id: "root"       name: "System Administrator"    rest: "/var/root:/bin/sh"
id: "daemon"     name: "System Services"         rest: "/var/root:/usr/bin/false"

rargs allow you to specify the delimiter (regex) to split the input on, and allows you to refer to the corresponding fields or field ranges. This allows it to be used as an AWK replacement for some simple but common cases.

How does it work?

receive the input on stdin and split it into lines
split (-d) or extract (-p) the input into named or numbered groups, with {0} matching the whole line
map the named and numbered groups into a command passed as the remaining arguments, and execute the command

Features

Regexp captures

rargs allows you to use any regular expression to match the input, and captures anything you are interested in. The syntax is the standard, mostly Perl-compatible Rust regex syntax used by tools such as ripgrep.

positional (numbered) groups are captured with parentheses, e.g. '(\w+):(\d+)', and the corresponding groups are referred to by {1}, {2} etc. in the command
named groups are captured with (?P<name>...) and referred to by {name} in the command

Delimiter captures

For simple usage, you might not want to write the whole regular expression to extract parts of the line. All you want is to split the groups by some delimiter. With rargs you can achieve this by using the -d (delimiter) option.

Field ranges

We already know how to refer to captures by number ({1}) or by name ({name}). There are also cases where you might want to substitute multiple fields at the same time. rargs also supports this with field-range expressions.

Suppose we have already captured 5 groups representing the strings 1, 2, 3, 4 and 5

{..} gathers them all into 1 2 3 4 5 (note that they are separated by a space; this can be overridden by the -s option)
{..3} results in 1 2 3
{4..} results in 4 5
{2..4} results in 2 3 4
{3..3} results in 3

You can also specify a "local" separator (which will not affect the global setting):

{..3:-} results in 1-2-3
{..3:/} results in 1/2/3

Negative field

Sometimes you may want to refer to the last few fields but have no way to predict the total number of fields of the input. rargs offer negative fields.

Suppose we have already captured 5 groups representing the strings 1, 2, 3, 4 and 5:

{-1} results in 5
{-5} results in 1
{-6} results in nothing
{-3..} results in 3 4 5

Multiple threading

You can run commands in multiple threads to improve performance:

-w <num> specifies the number of workers you want to run simultaneously
-w 0 defaults the number of workers to the number of CPUs on your system

Special Variables

{LINENUM} or {LN} to refer to current line number.

Interested?

All feedback and PRs are welcome!

rargs's People

Contributors

Stargazers

Watchers

rargs's Issues

Verbose mode

xargs has a -t argument which prints the commands before running it!

Rename threads option (`-w`) to `-j`

Many tools, such as GNU Parallel, GNU Make, fd, ripgrep, are using -j to specify the number of concurrents threads/jobs.

Adopting the same convention would make Rargs easier to grasp.

panic on missing cmd

Currently rargs completely panics if you pass it a program it can't exec; I'd expected it to fail of course, but maybe with a slightly better message 😄

$ cat myfile.txt | rargs something
thread '<unnamed>' panicked at 'command failed to start: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1165:5
...

It prints the panic for each line of the input file.

Thanks for the tool! I've been looking for something handy like this 👍

Wrong version number

Running rargs --version from rargs-v0.2.3-x86_64-unknown-linux-gnu.tar.gz gives 0.2.2.

Default to using multi-threading

For the sake of ergonomics and performance, IMO the default number of threads should be the number of CPU's. In my experience, the cases where single-threaded behavior is required are the exception, not the rule, and users can explicitly opt in for single-threaded behavior if needed.

This would indirectly solve a different issue: the only way to automatically peg the number of threads to the number of CPU's is to set --worker 0 --threads 0 even though the worker argument is listed as deprecated.

argument without {}

now:
cmd | rargs rm {}

I want not to have to specify: {}
i want to do:
cmd | rargs rm
without {}

you can check if 1 argument then you can omit: {}

Unit test failed on Windows

Ran cargo test for testing unit test and two tests failed on a Windows machine.

The error of a cargo test looks like:

running 15 tests
test test_field_separator ... FAILED
test regex_should_match ... FAILED
test test_default_delimiter ... FAILED
test test_delimiter ... FAILED
test test_global_field_separator ... FAILED
test test_line_num_should_work ... FAILED
test test_no_read0 ... ok
test test_negtive_regex_group_should_work ... FAILED
test test_range_both ... FAILED
test test_range_left_inf ... FAILED
test test_range_right_inf ... FAILED
test test_read0_long ... FAILED
test test_read0_short ... FAILED
test test_regex_group_name_should_match ... FAILED
test test_start_num_should_be_working ... FAILED

failures:

---- test_field_separator stdout ----
thread 'test_field_separator' panicked at 'Assertion failed for ./target/release/rargs -d , echo X{3..4:_}X
with: Unexpected stdout

with: Didn't match.
diff=
``-X3_4X

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- regex_should_match stdout ----
thread 'regex_should_match' panicked at 'Assertion failed for `./target/release/rargs -p ^(?P<year>\d{4})-(\d{2})-(\d{2})$ echo {1} {2} {3}`
with: Unexpected stdout

with: Didn't match.
diff=
``-2018 01 20
```', C:\Users\Administrator\.cargo\registry\src\index.crates.io-6f17d22bba15001f\assert_cli-0.6.3\src\assert.rs:441:13

---- test_default_delimiter stdout ----
thread 'test_default_delimiter' panicked at 'Assertion failed for `./target/release/rargs echo X{1},{2},{3}X`
with: Unexpected stdout

with: Didn't match.
diff=
``-Xa,b,cX
```', C:\Users\Administrator\.cargo\registry\src\index.crates.io-6f17d22bba15001f\assert_cli-0.6.3\src\assert.rs:441:13

---- test_delimiter stdout ----
thread 'test_delimiter' panicked at 'Assertion failed for `./target/release/rargs -d , echo X{1},{2},{3},{4}X`
with: Unexpected stdout

with: Didn't match.
diff=
``-Xa,b,,cX
```', C:\Users\Administrator\.cargo\registry\src\index.crates.io-6f17d22bba15001f\assert_cli-0.6.3\src\assert.rs:441:13

---- test_global_field_separator stdout ----
thread 'test_global_field_separator' panicked at 'Assertion failed for `./target/release/rargs -d , -s / echo X{3..4}X`
with: Unexpected stdout

with: Didn't match.
diff=
``-X3/4X
```', C:\Users\Administrator\.cargo\registry\src\index.crates.io-6f17d22bba15001f\assert_cli-0.6.3\src\assert.rs:441:13

---- test_line_num_should_work stdout ----
thread 'test_line_num_should_work' panicked at 'Assertion failed for `./target/release/rargs echo {LN} {}`
with: Unexpected stdout

with: Didn't match.
diff=
``-1 line 1
2 line 2
```', C:\Users\Administrator\.cargo\registry\src\index.crates.io-6f17d22bba15001f\assert_cli-0.6.3\src\assert.rs:441:13

---- test_negtive_regex_group_should_work stdout ----
thread 'test_negtive_regex_group_should_work' panicked at 'Assertion failed for `./target/release/rargs -p ^(?P<year>\d{4})-(\d{2})-(\d{2})$ echo {-3} {-2} {-1}`
with: Unexpected stdout

with: Didn't match.
diff=
``-2018 01 20
```', C:\Users\Administrator\.cargo\registry\src\index.crates.io-6f17d22bba15001f\assert_cli-0.6.3\src\assert.rs:441:13

---- test_range_both stdout ----
thread 'test_range_both' panicked at 'Assertion failed for `./target/release/rargs -d , echo X{3..3}X`
with: Unexpected stdout

with: Didn't match.
diff=
``-X3X
```', C:\Users\Administrator\.cargo\registry\src\index.crates.io-6f17d22bba15001f\assert_cli-0.6.3\src\assert.rs:441:13

---- test_range_left_inf stdout ----
thread 'test_range_left_inf' panicked at 'Assertion failed for `./target/release/rargs -d , echo X{..3}X`
with: Unexpected stdout

with: Didn't match.
diff=
``-X1 2 3X
```', C:\Users\Administrator\.cargo\registry\src\index.crates.io-6f17d22bba15001f\assert_cli-0.6.3\src\assert.rs:441:13

---- test_range_right_inf stdout ----
thread 'test_range_right_inf' panicked at 'Assertion failed for `./target/release/rargs -d , echo X{3..}X`
with: Unexpected stdout

with: Didn't match.
diff=
``-X3 4 5 6X
```', C:\Users\Administrator\.cargo\registry\src\index.crates.io-6f17d22bba15001f\assert_cli-0.6.3\src\assert.rs:441:13

---- test_read0_long stdout ----
thread 'test_read0_long' panicked at 'Assertion failed for `./target/release/rargs --read0 echo {}`
with: Unexpected stdout

with: Didn't match.
diff=
``-a
b
```', C:\Users\Administrator\.cargo\registry\src\index.crates.io-6f17d22bba15001f\assert_cli-0.6.3\src\assert.rs:441:13

---- test_read0_short stdout ----
thread 'test_read0_short' panicked at 'Assertion failed for `./target/release/rargs -0 echo {}`
with: Unexpected stdout

with: Didn't match.
diff=
``-a
b
```', C:\Users\Administrator\.cargo\registry\src\index.crates.io-6f17d22bba15001f\assert_cli-0.6.3\src\assert.rs:441:13

---- test_regex_group_name_should_match stdout ----
thread 'test_regex_group_name_should_match' panicked at 'Assertion failed for `./target/release/rargs -p ^(?P<year>\d{4})-(\d{2})-(\d{2})$ echo {year} {2} {3}`
with: Unexpected stdout

with: Didn't match.
diff=
``-2018 01 20
```', C:\Users\Administrator\.cargo\registry\src\index.crates.io-6f17d22bba15001f\assert_cli-0.6.3\src\assert.rs:441:13

---- test_start_num_should_be_working stdout ----
thread 'test_start_num_should_be_working' panicked at 'Assertion failed for `./target/release/rargs -n 10 echo {LN} {}`
with: Unexpected stdout

with: Didn't match.
diff=
``-10 line 1
11 line 2
```', C:\Users\Administrator\.cargo\registry\src\index.crates.io-6f17d22bba15001f\assert_cli-0.6.3\src\assert.rs:441:13


failures:
    regex_should_match
    test_default_delimiter
    test_delimiter
    test_field_separator
    test_global_field_separator
    test_line_num_should_work
    test_negtive_regex_group_should_work
    test_range_both
    test_range_left_inf
    test_range_right_inf
    test_read0_long
    test_read0_short
    test_regex_group_name_should_match
    test_start_num_should_be_working

test result: FAILED. 1 passed; 14 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.12s


** rargs version - Rargs 0.3.0 **
** OS version - Windows version 10.0.20348 Build 20348 **

Allow to only run command when given pattern is matched

Hi and happy new year,

Given the most general running scenario involving a pattern

rargs -p <pattern> <cmd>

I'm unsure whether it makes sense to run cmd no matter if the pattern is matched ; but at least I can tell that it is not always what I expect from Rargs.

Take for instance the common scenario when some files, e.g.

1. Some name
10. Another name

need to be renamed like

01. Some name
10. Another name

There, I wish I was able to type

ls |rargs -p '^(\d\..*)' -- mv "'{1}'" "'0{1}'"

but it wouldn't currently work since Rargs would execute mv for both files, instead of the first one only.

Exit code?

Firstly — thank you v much for rargs! I use it a lot. It's probably the most underrated "new cli tools" library.

What should the exit code of an rargs command?

Currently it seems to return successfully even if there were lots of errors in the individual commands.

I'd like to be able to understand if any command failed (e.g. did any string fail to be grepped in a file). Is there a way of doing that with the current library?