fd0 / machma Goto Github PK
View Code? Open in Web Editor NEWEasy parallel execution of commands with live feedback
License: BSD 2-Clause "Simplified" License
Easy parallel execution of commands with live feedback
License: BSD 2-Clause "Simplified" License
$ cat foo.sh
#!/bin/sh
echo $1
$ for a in {1..10}; do for b in {1..100}; do echo $b; done | go run github.com/fd0/machma@latest -- ./foo.sh "{}" | wc -l; done
95
101
101
101
99
102
100
101
100
100
One would expect the same number of output lines on each run: 100.
Comparing two runs:
$ doit() { sort <(for b in {1..100}; do echo $b; done | go run github.com/fd0/machma@latest --no-id --no-timestamp -- ./foo.sh "{}") }
$ diff -U0 <(doit) <(doit)
--- /proc/self/fd/11 2021-12-31 02:37:51.409602436 +0100
+++ /proc/self/fd/12 2021-12-31 02:37:51.409602436 +0100
@@ -30,0 +31 @@
+35 35
@@ -33 +33,0 @@
-38 38
@@ -52,0 +53,2 @@
+56 56
+57 57
@@ -66,0 +69 @@
+71 71
It would be nice if the output could be controlled by using options.
The worker number and timestamp either contain data I don't need or are already provided by the data being output from my other process. These could be "off' by default and enabled with options or "on" by default and disabled with options.
It might also be nice to have an argument that accepts a keyword to act as a filter. Grepping is so easy that maybe it wouldn't be worth the effort but then the nice colorized output is lost.
Here's an example of how I would like to use machma
to replace parallel in my environment.
find . -name "*.png" | machma -- convert {} {}.jpg
Another way I've been experimenting with quotes and slashes to see if I get it to be interpreted correctly is by feeding it to echo
.
find . -name "*.png" | machma -- echo {} {}.jpg
The expected result is that the original file would remain in place and a new one with the ".jpg" extension would be created.
What I get is a single file named "{}.jpg".
The number of lines in the status area is variable so reading and keeping track of the processed/failed/workers area is difficult.
An example to see what I'm talking about is to run find / | machma -p 9 -- ls -alh {}
and try to keep an eye on the "processed" count.
Maybe there's a better solution but what I was thinking: adding empty lines so that the status area stays in the same location. So there would always be number of lines equal to the maximum workers count plus one.
For example:
echo "a.jpg b.jpg" | machma -- cp $1 $2
I'm trying to use machma to speed up some operations in build scripts, but it always seems to hang without a TTY (which is the case when my build scripts normally run). I have reproduced this behavior inside of a docker container simply by including or omitting the -t
flag.
This will hang:
$ docker run --rm golang:1.9 /bin/bash -c 'go get -u github.com/fd0/machma ; find /go -iname "*.go" | machma -- md5sum {}'
But this will not hang:
$ docker run -t --rm golang:1.9 /bin/bash -c 'go get -u github.com/fd0/machma ; find /go -iname "*.go" | machma -- md5sum {}'
The only difference in the commands is the -t
flag.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.