Giter VIP home page Giter VIP logo

zt-exec's Introduction

ZT Process Executor

Continuous Integration

Build Status

Quick Overview

The project was created to merge similar functionality of projects at ZeroTurnaround into a single codebase. It's designed to be powerful but still simple to use. By using a single class ProcessExecutor the user gets the functionality from both java.lang.ProcessBuilder and Apache Commons Exec.

Dependencies

  • Minimal required Java version is 6 (tested also with 8, 11 and 17).
  • The only 3rd-party dependency is SLF4J.

Installation

Maven Central

The project artifacts are available in Maven Central Repository.

To include it in your maven project then you have to specify the dependency.

...
<dependency>
    <groupId>org.zeroturnaround</groupId>
    <artifactId>zt-exec</artifactId>
    <version>1.12</version>
</dependency>
...

Motivation

There are many approaches to take when running external processes from Java. There are the JRE options such as the Runtime.exec() and ProcessBuilder. Also there is the Apache Commons Exec. Nevertheless we created yet another process library (YAPL).

Some of the reasons for this crazy endeavour

  • Improved handling of streams
  • Reading/writing to streams
  • Redirecting stderr to stdout
  • Improved handling of timeouts
  • Improved checking of exit codes
  • Improved API
  • One liners for quite complex usecases
  • One liners to get process output into a String
  • Access to the Process object available
  • Support for async processes ( Future )
  • Improved logging with SLF4J API
  • Support for multiple processes

Can zt-exec also kill processes?

No. There is zt-process-killer for that.

Examples

  • Output is pumped to NullOutputStream
new ProcessExecutor().command("java", "-version").execute();

  • Returning the exit code
  • Output is pumped to NullOutputStream
int exit = new ProcessExecutor().command("java", "-version")
                  .execute().getExitValue();

  • Return output as UTF8 String
String output = new ProcessExecutor().command("java", "-version")
                  .readOutput(true).execute()
                  .outputUTF8();    

  • Pumping the output to a logger
new ProcessExecutor().command("java", "-version")
      .redirectOutput(Slf4jStream.of(LoggerFactory.getLogger(getClass().getName() + ".MyProcess")).asInfo()).execute();

  • Pumping the output to a logger (short form for previous)
new ProcessExecutor().command("java", "-version")
      .redirectOutput(Slf4jStream.of("MyProcess").asInfo()).execute();

  • Pumping the output to the logger of the caller class
new ProcessExecutor().command("java", "-version")
      .redirectOutput(Slf4jStream.ofCaller().asInfo()).execute();

  • Pumping the output to a logger
  • Returning output as UTF8 String
String output = new ProcessExecutor().command("java", "-version")
        .redirectOutput(Slf4jStream.of(getClass()).asInfo())
        .readOutput(true).execute().outputUTF8();

  • Pumping the stderr to a logger
  • Returning the output as UTF8 String
String output = new ProcessExecutor().command("java", "-version")
        .redirectError(Slf4jStream.of(getClass()).asInfo())
        .readOutput(true).execute()
        .outputUTF8();

  • Running with a timeout of 60 seconds
  • Output pumped to NullOutputStream
try {
  new ProcessExecutor().command("java", "-version")
        .timeout(60, TimeUnit.SECONDS).execute();
}
catch (TimeoutException e) {
  // process is automatically destroyed
}

  • Pumping output to another OutputStream
OutputStream out = ...;
new ProcessExecutor().command("java", "-version")
      .redirectOutput(out).execute();

  • Handling output line-by-line while process is running
new ProcessExecutor().command("java", "-version")
    .redirectOutput(new LogOutputStream() {
      @Override
      protected void processLine(String line) {
        ...
      }
    })
    .execute();

  • Destroy the running process when VM exits
  • Output pumped to NullOutputStream
new ProcessExecutor().command("java", "-version").destroyOnExit().execute();

  • Run process with a specific environment variable
  • Output pumped to NullOutputStream
new ProcessExecutor().command("java", "-version")
    .environment("foo", "bar").execute();

  • Run process with a specific environment
  • Output pumped to NullOutputStream
Map<String, String> env = ...
new ProcessExecutor().command("java", "-version")
    .environment(env).execute();

  • Throw exception when wrong exit code
  • Output is pumped to NullOutputStream
try {
  new ProcessExecutor().command("java", "-version")
        .exitValues(3).execute();
}
catch (InvalidExitValueException e) {
  System.out.println("Process exited with " + e.getExitValue());
}

  • Throw exception when wrong exit code
  • Return output as UTF8 String
String output;
boolean success = false;
try {
  output = new ProcessExecutor().command("java", "-version")
                .readOutput(true).exitValues(3)
                .execute().outputUTF8();
  success = true;
}
catch (InvalidExitValueException e) {
  System.out.println("Process exited with " + e.getExitValue());
  output = e.getResult().outputUTF8();
}

  • Starting process in the background
  • Output is pumped to NullOutputStream
Future<ProcessResult> future = new ProcessExecutor()
                                    .command("java", "-version")
                                    .start().getFuture();
// do some stuff
future.get(60, TimeUnit.SECONDS);

  • Start process in the background
  • Return output as UTF8 String
Future<ProcessResult> future = new ProcessExecutor()
                                    .command("java", "-version")
                                    .readOutput(true)
                                    .start().getFuture();
// do some stuff
String output = future.get(60, TimeUnit.SECONDS).outputUTF8();

zt-exec's People

Contributors

aalmiray avatar guw avatar johnlcox avatar ketan avatar madisparn avatar paulwoitaschek avatar qxo avatar rnc avatar schoepke avatar sullis avatar swimmesberger avatar toomasr avatar tsjensen avatar villane avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

zt-exec's Issues

End of process interruption breaks if redirected InputStream swallows interruption

When connecting a process' standard in using ProcessExecutor.redirectInput(InputStream) to another InputStream that blocks on read, the interruption behavior depends on the underlying InputStream.

In my example I had an InputStream which swallowed InterruptedException. Thus, when the process exited, the pumping thread is not interrupted in is.read(buf) (line 168, StreamPumper.java). Well, the interruption happens within is.read(buf) and if this decides to ignore the interruption then the pumping thread does not die.

In my particular case, the InputStream is coming from a console. Thus, it is not closing and waiting on a user to input further characters.

Looking at the implementation it seems that it relies on proper interruption causing a finish of the pumping thread. This is not happening in my environment. I'll try to come up with a test case for reproducing.

An error InvalidExitValueException while executing the unix command

When I execute the ubuntu command through suno, an exception will throws.
String output = new ProcessExecutor().command("sudo","touch","/opt/cc") .redirectInput(new ReaderInputStream(new StringReader("mypassword\n"))) .timeout(30, TimeUnit.SECONDS) .readOutput(true) .exitValues(0) .execute() .getOutput() .getUTF8();

The exception is as follows
org.zeroturnaround.exec.InvalidExitValueException: Unexpected exit value: 1, allowed exit values: [0], executed command [sudo, touch, /opt/cc], output was 54 bytes: sudo: no tty present and no askpass program specified

I know that you can modify sudoers.d so that users do not have to enter a password.
But I do not want to do this, change the system's security policy.
Thank you!

Need support for MDC logging context

When we use a logger to redirect process output, it loses any MDC logging context from the provided logger.
Currently ZT launches a new executor and a thread to execute the process. There is no way to set or pass the logging context to the process execution thread. This is a big pain point when we use ZT to execute processes in cloud where multiple threads, processes and machines are involved. Very hard to debug anything without a logging context.
Easiest way to do this in current zt code is to invoke one of the before listeners from the process thread and not from the calling thread. Alternate way is to provide hooks into the thread executor creation.
Sample code to show the problem:
`
import java.util.Map;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;
import org.zeroturnaround.exec.ProcessExecutor;
import org.zeroturnaround.exec.stream.slf4j.Slf4jStream;

public class Main {
private static final Logger logger = LoggerFactory.getLogger(Main.class);

public static void main(String[] args) throws Exception {
	MDC.put("jobId", "12232");
	logger.info("Starting ls process");
	ProcessExecutor pe = new ProcessExecutor("ls", "-l");
	pe.redirectOutput(Slf4jStream.of(logger).asInfo());
	pe.execute();
	final Map<String,String> ctxMap = MDC.getCopyOfContextMap();
	new Thread(){
		public void run() {
			MDC.setContextMap(ctxMap);
			logger.info("Log from a child thread");
		};
	}.start();
}

}
`

Log output from the above code. Notice how the process output does not have the context info (jobId):

[main] [jobId=12232] INFO ztlogmdctest.Main - Starting ls process [main] [jobId=12232] DEBUG o.z.exec.ProcessExecutor - Executing [ls, -l]. [main] [jobId=12232] DEBUG o.z.exec.ProcessExecutor - Started java.lang.UNIXProcess@5b464ce8 [Thread-0] [] INFO ztlogmdctest.Main - total 8 [Thread-0] [] INFO ztlogmdctest.Main - -rw-r--r-- 1 maruthir staff 831 Jan 27 11:42 pom.xml [Thread-0] [] INFO ztlogmdctest.Main - drwxr-xr-x 4 maruthir staff 136 Jan 27 11:36 src [Thread-0] [] INFO ztlogmdctest.Main - drwxr-xr-x 4 maruthir staff 136 Jan 27 11:36 target [main] [jobId=12232] DEBUG o.zeroturnaround.exec.WaitForProcess - java.lang.UNIXProcess@5b464ce8 stopped with exit code 0 [Thread-1] [jobId=12232] INFO ztlogmdctest.Main - Log from a child thread

Error with handling output line-by-line while process is running code snippet

I'm getting below error while trying out "Handling output line-by-line while process is running".

Error:
Error:(27, 33) java: constructor LogOutputStream in class sun.rmi.log.LogOutputStream cannot be applied to given types;
required: java.io.RandomAccessFile
found: no arguments
reason: actual and formal argument lists differ in length

Test Code
import org.zeroturnaround.exec.ProcessExecutor;
import sun.rmi.log.LogOutputStream;
import java.io.IOException;

public class TestCmd {
public static void main(String[] args) throws IOException, InterruptedException {
new ProcessExecutor().command("java", "-version")
.redirectOutput(new LogOutputStream() {
@OverRide
protected void processLine(String line) {
System.out.println("hereeee" + line);
}
})
.execute();
}
}

Question: RedirectOutput(CachingOutputStream)

Hello,

I am using zt-exec to execute a process in background and read the output line by line with the LogOutputStream.processLine() method.

Unfortunately, when doing some complex stuff in processLine it seems to miss some output lines, can someone give me an advice how to cache/buffer the lines, to process them later?

Thanks for your work!

Problem with running a command with an environment variable.

I couldn't find a test for this case, so I come up with my test case

    @Ignore("This test constantly fails")
    @Test
    public void testDefineEnvironmentVariable() throws IOException, InterruptedException, TimeoutException {
        String output = processExecutor
                .command("echo", "$foo")
                .environment("foo", "bar")
                .readOutput(true).execute()
                .outputUTF8();

        assertThat(output, containsString("bar"));
    }

Output:
screenshot from 2018-06-23 14-02-24

I expect from method environment() to add a variable into environment.
Java .environment("foo", "bar") should be console equals export foo=bar
So now echo $foo should return bar, but in the test I got $foo.

Can not get output result line by line

The results come out after main.py run out instand of line by line. Is there anything wrong for my code? Thanks!

java code

new ProcessExecutor().command("python", "/the/path/to/main.py")
                .redirectOutput(new LogOutputStream() {
                    @Override
                    protected void processLine(String line) {
                        System.out.println(line);
                    }
                })
                .execute();

main.py

import time

if __name__ == "__main__":
  for i in range(0, 10):
    print ('hello: '+str(i))
    time.sleep(1)

Is there a way to set the process priority

I really like the simplicity of your code, I have a test app written that works quite well with it. The one thing that I am unable to do, is to set the priority of the executed process. In the past, I used the Java ProcessBuilder and had platform specific code to use nice on Unix/Linux/Mac and use START /LOW on Windows.

I could still use START/LOW on Windows, but the problem I was trying to solve that brought me to trying this library, was that of the process tree getting disconnected. I want to have it be the case that closing the Java VM (CTRL-C into a CMD window which is launched from the command line using java -jar) will also cause the background tasks to shutdown. My current approach is unable to manage this, but I was hoping using this library could allow it... but I have to have the processes running at the lowest possible priority, as they are long running tasks attempting to consume only idle cycles and I need my machine to otherwise stay responsive.

zt-exec converting multi-line output to single line

Hi,

I am facing a problem that is my output, which is being redirected to logger is quite large and it's multi-line but zt-exec converts that multi-line to a single line. Due to this reason my business logic is failing. Is there a way in zt-exec to maintain the new lines and carriage returns.

Closing a Process InputStream hangs on some processes

(1) I noticed that you have this is StandardProcessCloser and what I find is that getInputStream() can hang when Process.destroy() has already been called:

try {
process.getInputStream().close();
}catch (IOException e) {
log.error("Failed to close process input stream:", e);
caught = e;
}

(2) Also, I find that InputStream.close() can hang, unless process.destroy() is called first.

So, the key for me is to make sure I obtain the inputStream before calling close() and then also make sure to call destroy before forcibly closing the streams:

For example, here is a process that I think requires that correct sequence or it will not destroyForcibly on Windows:

@echo OFF
ECHO WELCOME TO HUNG
:loop
for /l %%n in (1,1,10000000000) do set temp=
goto:loop
ECHO UNREACHABLE
EXIT

Furthermore, if the Process is started as a 16-bit DOS executable, then even though Process reports that it is dead, it is still alive and well in the ntvdm.exe subsystem for running windows-on-windows 16-bit exe's in the system32 environment.

Which is to say, that although the batch file above needs a specific order to shutdown, if it spawns a 16-but program then that program must be forcibly killed by taskkill.

Run command for change directory and then git clone using zt-exec

I am trying the following code :

String output = new ProcessExecutor().command("cd", "D:/myfolder")
            .readOutput(true).execute()
            .outputUTF8();  

But getting an exception :
Exception in thread "main" java.io.IOException: Could not execute [cd D:/myfolder].
at org.zeroturnaround.exec.ProcessExecutor.invokeStart(ProcessExecutor.java:936)
at org.zeroturnaround.exec.ProcessExecutor.startInternal(ProcessExecutor.java:910)
at org.zeroturnaround.exec.ProcessExecutor.execute(ProcessExecutor.java:860)
at .git.zttest.main(zttest.java:16)
Caused by: java.io.IOException: Cannot run program " cd D:/myfolder": CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessBuilder.start(Unknown Source)
at org.zeroturnaround.exec.ProcessExecutor.invokeStart(ProcessExecutor.java:931)
... 3 more
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
... 5 more

How this command can be executed using zt-exec.
Also after this is done my whole command is :
cd <mydirectory> && D: && <gitlocation>/git clone --depth 1 <url>

Is this possible?

A few questions..

A few questions:

  • Does the API support killing an async process?
  • Can you get access to the PID? If the process spawns a child can you get access to them? eg. output, PIDs, ability to manage/kill them?

Thanks.

Long Output is not being printed to the logger

Hi,

I have some chef nodes which I want to print to logger but the nodes are not printed. Although other output like a single value from node or my debug comments are printed but not the entire node.

private static final Logger logger = Logger.getLogger(MyProcessRunner.class);
 OutputHandler outputStream = new OutputHandler(logger);
 ErrorHandler errorStream = new ErrorHandler(logger);

new ProcessExecutor().command(cmd).environment(env).directory(workingDir)
          .redirectOutput(new LogOutputStream() {
            @Override
            protected void processLine(String line) {
              outputStream.WriteOutputToLogger(line);
            }
          }).redirectError(new LogOutputStream() {
            @Override
            protected void processLine(String line) {
              errorStream.WriteOutputToLogger(line);
            }
          }).timeout(timeoutInSeconds, TimeUnit.SECONDS).execute()

The output stream has a method to handle data before printing to logger like:

 public void WriteOutputToLogger(String line) {
        if (rowCount < maxRowCount * 2) {

            logger.info(logKey + "cmd error: " + line);
            result.appendStdErr(line + "\n");

        } else if (rowCount == maxRowCount * 2) {
            logger.warn(logKey
                    + " hit max amount of output per process of "
                    + maxRowCount
                    + " lines. Please run the workorder on the box: chef-solo -c /home/chef/cookbooks/chef.rb -j /opt/chef/workorder/someworkorder ");
        }
        rowCount++;
    }

Test zt-exec command line escaping

Test what happens with spaces and quote marks both on Win and UNIX.
Compare the result with Commons Exec CommandLine.
After that let's discuss how to handle command line in zt-exec better.

Force destruction of a known PID

For Windows... We have some processes that __ just __ won't __ die -- in particular, there are 16-bit processes that are run in the ntvdm.exe subsystem and even thought the Process.destroyForcibly() reports that they are dead, they are still alive because Java only kills the consolehost or cmd.exe not the spawned thread in ntdvm.exe.

Also, getting the exit value fails (or throws) because the process is still alive even after Process claims it died.

So ... I have my own library for Process that scans the process output for a self reporting PID.

If the process exit is overdue, then I call my taskkiller ... and it goes through all the PID's found and kills them all via the windows taskkill command.

Hence, all my processes are started via a 'cmd /c' batchfile that calls getpid.exe and then reports the errorlevel as the PID like this:

%BATCHOME%\manage\getpid.exe
echo GETPID.EXE: THE CURRENT PID IS %errorlevel%
:: then call program that does what we need done.
:: by running "taskkill.exe /f /t /pid {pid}" we can kill the PID and any children in the tree

The getpid.exe program is created by running the batchfile code pasted at the bottom of this post.

I'll throw this out there and then maybe you can integrate a way to collect PID's using some program specified magic cookie and then be able to verifiably destory them similar to this.

The PID's are collected by my process reader. Here is the code to my process destroyer util.

public static List<Throwable> kill(String pid, Process p) throws InterruptedException {
    List<Throwable> exceptions = new ArrayList();
            try {
                callWindowsTaskkill(p, pid, exceptions);
            } catch (final KillException ke) {
                exceptions.add(ke);
            }
    return exceptions;
}

private static void destroyWait(Process p) throws InterruptedException, KillException {
    p.destroyForcibly();
    logger.debug("destroyForcibly called. Will wait 100 ms for official termination.");
    if (p.waitFor(100, TimeUnit.MILLISECONDS)) {
        logger.info(">> exit code: {}", p.exitValue());
    } else {
        throw new KillException("Process will not die after calling destroyForcefully() and waiting 100ms.", p);
    }
}

private static void callWindowsTaskkill(Process p, String pid, List<Throwable> exceptions) throws InterruptedException, KillException {
    logger.info("calling kill on pid: {}", pid);
    BatchSession<LineGobbler> bs = getKillTask(pid).call();
    for (Throwable t : bs.getExceptions()) {
        exceptions.add(t);
    }
    if (bs.getGobbler().isPresent()) {
        boolean dead = false;
        for (String line : bs.getGobbler().get().getLines()) {
            logger.debug(line);
            if (StringUtils.startsWithIgnoreCase(line, "SUCCESS: The process with PID " + pid + " ")
                    && StringUtils.endsWithIgnoreCase(line, " has been terminated.")) {
                dead = true;
            } else if (StringUtils.containsIgnoreCase(line, "ERROR: The process \"" + pid + "\" not found")) {
                dead = true;
            }
        }
        if (!dead) {
            throw new KillException("Process with pid " + pid + " will not die after being destroyed by 'taskkill'", p);
        }
    }
}

private static BatchTask<LineGobbler> getKillTask(String pid) {
    String id = pid + "-killer";
    return BatchTask.builder()
            .setTimeout(Duration.ofSeconds(10))
            .setTimeoutAction(BatchTask.TimeoutAction.taskkill)
            .build(id, new ProcessBuilder("taskkill.exe", "/f", "/t", "/pid", pid),
                    () -> new LineGobbler(id, StandardCharsets.ISO_8859_1));
}

@echo Off
Rem Script made using BHX { consolesoft.com/p/bhx }
SetLocal EnableExtensions EnableDelayedExpansion
Set "bin=getpid.ex_"
Set "size=643"
For %%# In (
"getpid.exe"
"!bin!" "!bin!.da" "!bin!.tmp"
) Do If Exist "%%#" (Del /A /F /Q "%%#" >Nul
If ErrorLevel 1 Exit /B 1 )
Findstr /B /N ":+res:!bin!:" "%~f0" >"!bin!.tmp"
(Set /P "inioff=" &Set /P "endoff=") <"!bin!.tmp"
For /F "delims=:" %%# In ("!inioff!") Do Set "inioff=%%#"
For /F "delims=:" %%# In ("!endoff!") Do Set "endoff=%%#"
Set ".=set o=createobject(#scripting.filesystemobject#)"
Set ".=!.!: set w=createobject(#adodb.stream#)"
Set ".=!.!: set r=o.opentextfile(#%~f0#,1,0,0)"
Set ".=!.!: w.charset=#windows-1252# : w.type=2 : w.open"
Set ".=!.!: for i=1 to !inioff! step 1 : r.readline : next"
Set ".=!.!: do while i<!endoff! : d=r.readline"
Set ".=!.!: for j=1 to len(d) step 2"
Set ".=!.!: w.writetext chr(cbyte(#&h#&mid(d,j,2)))"
Set ".=!.!: next : i=i+1 : loop"
Set ".=!.!: w.savetofile #!bin!#,2"
Set ".=!.!: w.close : r.close"
Set ".=!.!: set w=nothing : set r=nothing : set o=nothing"
Set ".=!.:#="!"
Echo !.!>"!bin!.da"
Set "ret=1"
Cscript.exe /B /E:vbs "!bin!.da" >Nul
For %%# In ("!bin!") Do If "%%~z#"=="!size!" Set "ret=0"
If "0"=="!ret!" Expand.exe -r "!bin!" -F:* . >Nul
If ErrorLevel 1 Set "ret=1"
Del /A /F "!bin!" "!bin!.da" "!bin!.tmp" >Nul
Exit /B !ret!

:+res:getpid.ex_:
4D5343460000000083020000000000002C000000000000000301010001000000
000000004700000001000100000800000000000000009645357D200067657470
69642E65786500DD597C9934020008434BF38D9AC0C0CCC0C0C002C4FFFF3330
EC6080000706C2A00188F9E477F1316CE13CABB883D1E7AC62484666B1424151
7E7A5162AE4272625E5E7E894252AA4251699E42669E828B7FB0426E7E4AAA1E
2F2F970AD48C005706061F466686A7A9334260E63E60E067E66664126760823A
0C0836080009010684EB406C268834234219C4E140F1E21606B0BF181814A0FA
04E0FA05903D61C0C0F08708BF920A5C80E6CAE091D72B49AD2801B9859101EE
173046020A0C01097A45298925890C0C2250AF208709C2CB060E7A9910757FA0
7E02AB63C3507700DD1DA19D4FC3C3821BDFF800D51F7751019B7C9C4505A4FB
C50620D179B8F90D87E181E6DFFF4B225E2C070AF46EDDF3F7FFFFCE1215964E
1995E3AD208E0E5075E7F1171381B240C5AD074A455F5B4015BC68810996A85A
EE3D02545CFA14245C0A14EE6EBD02E437BF61E94DFD121D171F7BB8B7448D01
24190194045A892AD1BB471DEADEE637122F5CFF03DDC0A2F2C21428340119FC
57F530706000D101503A044A4740691728ED03A1FF83523D28ED43E89106DC9D
9DAD14344A5272758D34154CF42CF40C4762288C5CA0618060BF3180945BD840
02503C0F88BB80780610AF03E2030684E582189C73F28B533D12F352725219F6
303817A52696A486E4E7E764A4E614181B05E725161467E4038B427146D78ACC
9280A2FCE4D4E26286FD8CEEA925CEA54545A9793031CF14864E6628DBD8C82D
B3A8B8249CA11B21E2072C51C361653A36ECED1AE4E7EA636CA4979293331AF3
100000
:+res:getpid.ex_:

TimeoutException is never thrown.

Hello, I tried run original test for TimeoutException https://github.com/zeroturnaround/zt-exec/blob/master/src/test/java/org/zeroturnaround/exec/test/ProcessExecutorTimeoutTest.java but it failed.

Also I developed simpler test to make it more readable:

@Test
public void testExecuteTimeout() throws Exception {
    try {
        new ProcessExecutor()
                .command(Lists.newArrayList("sleep","3"))
                .timeout(1, TimeUnit.SECONDS)
                .execute();
        Assert.fail("TimeoutException expected.");
    } catch (TimeoutException e) {
        Assert.assertThat(e.getMessage(), CoreMatchers.containsString("1 second"));
    }
}

@Test
public void testStartTimeout() throws Exception {
    try {
        new ProcessExecutor()
                .command(Lists.newArrayList("sleep","3"))
                .start()
                .getFuture()
                .get(1, TimeUnit.SECONDS);
        Assert.fail("TimeoutException expected.");
    } catch (TimeoutException e) {
        Assert.assertNull(e.getMessage());
    }
}

This test also failed.

The reason for all fallen tests is the TimeoutException is never thrown.

sub-processor never stop !

hi, i use zt-exec to run "ffmpeg" commands in my java application. and i set the timeout of the executor like code below:
new ProcessExecutor().command(cmds.split(" ")).timeout(TIMEOUT, TimeUnit.SECONDS)

when the timeout occurs, it will throw a exception, but it never kill the ffmpeg processor. the ffmpeg still run in background!

plz help me !!

timeout doesn't kill child process

Environment: Windows7, JDK 1.7

Steps to reproduce:

  • Create a batch file which invokes a main() with a simple timeout:

    public class Sleeper {
      static Logger logger = Logger.getLogger(WriteToDb.class);
    
      public static void main(String[] args) throws SQLException, InterruptedException {
              Thread.sleep(20000);
      }
    }
    
    * Create an executor with a timeout of two seconds:
    ```java
    public class ZtInvoker {
    
      public int invokeConnection(int timeout) throws InvalidExitValueException,
              IOException, InterruptedException {
    
          try {
              ProcessResult processResult = new ProcessExecutor()
                      .command("../test.bat")
                      .timeout(timeout, TimeUnit.MILLISECONDS)
                      .redirectOutputAlsoTo(System.out)
                      .redirectErrorAlsoTo(System.err)
                      .destroyOnExit()
                      .execute();
              return processResult.getExitValue();
          } catch (TimeoutException e) {
              // process is automatically destroyed
              System.out.println("timed out!");
              return -1;
          }
    
      }
    }
  • call ZtInvoker

 public class TestZtInvoker {

    @Test
    public void testInvoke() throws InvalidExitValueException, IOException, InterruptedException {
        ZtInvoker ztInvoker = new ZtInvoker();
        ztInvoker.invokeConnection(4000);
    }
 }
  • Watch the unit test and the child processes in task manager or process explorer

Actual result:

  • The test finishes in 4 seconds with the message "timed out!"
  • The child process continues for the full 20 seconds.

Expected:

  • I expect the child process to be killed after 4 seconds. The documentation indicates as much.

Note:

  • tried different versions of this test including using a one second timer in a loop and writing to a log file the whole time. I can confirm that the process is still alive and doing work.

StreamPumper threads are not interrupted when main thread is interrupted

StreamPumper threads are not interrupted when main thread is interrupted.

This piece of code seems to miss an outputThread.interrupt().

if (outputThread != null) { log.trace("Joining output thread {}...", outputThread); try { outputThread.join(); outputThread = null; } catch (InterruptedException e) { // ignore } }

This cause threads to be blocked until the process dies. Or it is necessary to call several time interrupt with some delay in order to really cause the thread to be interrupted.

StreamPumper should regularly flush OutputStream

When running on Mac and connecting a process' standard in using ProcessExecutor.redirectInput(InputStream) on Java 8 a process may not receive any data until the OutputStream is flushed. The reason is that the launched process is a UNIXProcess which wraps the OutputStream for writing to a process' standard in into a BufferedOutputStream.

Proposal: call out.flush() immediately after writing any data to out in StreamPumper.run line 129.

Note, I have not verified this behavior on other platforms but according to this blog, Java seems to be doint it in general for anything using UNIXProcess.

Maven dependency for zt-exec is incomplete

Hi.

I just pulled in your nifty looking zt-exec dependency and noticed that ProcessExecutor is missing. In fact I uncompressed the zt-exec jar and a lot of classes from your git project are missing.

I'm not sure if I'm expected to include any other ZT dependencies, but zt-exec doesn't appear to have any. Thanks.

inflating: org/zeroturnaround/exec/ProcessResult.class
inflating: org/zeroturnaround/exec/StartedProcess.class
inflating: org/zeroturnaround/exec/stop/DestroyProcessStopper.class
inflating: org/zeroturnaround/exec/stop/NopProcessStopper.class
inflating: org/zeroturnaround/exec/stop/ProcessStopper.class
inflating: org/zeroturnaround/exec/stream/CallerLoggerUtil.class
inflating: org/zeroturnaround/exec/stream/ExecuteStreamHandler.class
inflating: org/zeroturnaround/exec/stream/InputStreamPumper.class
inflating: org/zeroturnaround/exec/stream/LogOutputStream.class
inflating: org/zeroturnaround/exec/stream/PumpStreamHandler.class
inflating: org/zeroturnaround/exec/stream/slf4j/Level.class
inflating: org/zeroturnaround/exec/stream/slf4j/Slf4jDebugOutputStream.class
inflating: org/zeroturnaround/exec/stream/slf4j/Slf4jErrorOutputStream.class
inflating: org/zeroturnaround/exec/stream/slf4j/Slf4jInfoOutputStream.class
inflating: org/zeroturnaround/exec/stream/slf4j/Slf4jOutputStream.class
inflating: org/zeroturnaround/exec/stream/slf4j/Slf4jStream$1.class
inflating: org/zeroturnaround/exec/stream/slf4j/Slf4jStream.class
inflating: org/zeroturnaround/exec/stream/slf4j/Slf4jTraceOutputStream.class
inflating: org/zeroturnaround/exec/stream/slf4j/Slf4jWarnOutputStream.class
inflating: org/zeroturnaround/exec/stream/StreamPumper.class
inflating: org/zeroturnaround/exec/WaitForProcess.class

Release latest master as new stable release

It has been a while since 1.9 and master now contains couple of small, but not insignificant improvements around logging.

Please consider bumping to 1.10 (or maybe 1.9.1) and making it available as stable release.

ProcessExecutor NoSuchFieldError: NULL_OUTPUT_STREAM

When I try to call the below class, I get java.lang.NoSuchFieldError: NULL_OUTPUT_STREAM and I'm not sure why. I've tried the various versions of Slf4jStream.ofClass()/of(getClass()), etc. This is written in groovy, and it's being called from my test suite. I'd be grateful for any suggestions as to what might be going wrong.

    class ProgramRunner{

	final Logger log = LoggerFactory.getLogger(getClass());
	

	public void runProcess(List<String> command, Map<String, String> environment, Long timeout){
		new ProcessExecutor().command(job.command)
		.redirectOutput(Slf4jStream.of(log).asInfo())
		.redirectError(Slf4jStream.of(log).asError())
		.timeout(command.timeout ?: Long.MAX_VALUE, TimeUnit.SECONDS)
		.environment(command.environment ?: new HashMap<String, String>())
		.execute();
	}
}

Add redirect*(File) methods

Add methods for redirecting stream to/from files (mimic ProcessBuilder methods in Java 7). Instead of only calling redirect_(_Stream) methods the file streams must be also closed automatically once the process is stopped. Also they shouldn't be opened unless the process is actually started.

Proper README

We need a readme file that states the motivation, the changes compared to Apache Commons Exec and also a getting started guide.

How to create a session

Hi, Is there way to use zt-exec in session based. Can I bind it to System.in and System.out continiously?

Handle start failures better

At least on UNIX it seems that the following exception may mean many things:
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:135)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
...

  1. File you're executing does not exist.
  2. Directory in which you're executing does not exist.
  3. File is a shell script with Windows line endings (\r\n) instead of UNIX ones.
    We could catch the IOException and check for those conditions to give better feedback.

ProcessBuilder#destroyOnExit clears other listeners

Hey, we observed a bug today after some minor refactoring. Turned out that someone changed a the code to invoke:

new ProcessExecutor().listener(MY_LISTENER).destroyOnExit()

instead of

new ProcessExecutor().destroyOnExit().listener(MY_LISTENER)

This lead to the listener MY_LISTENER being cleared from the list of listeners by invoking ProcessExecutor#listener which invokes ProcessExecutor#clearListeners Is this wanted behavior? It surprised us.

Stream closed exception on Java 1.8 when piping input stream

Using zt-exec 1.8 and Java 1.8 I'm getting the following error when trying to do echo foo | /bin/true:

2016-01-27 14:32:51,780 ERROR [Thread-0] [o.z.e.s.StreamPumper] Got exception while closing exhausted output stream
java.io.IOException: Broken pipe
        at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_71]
        at java.io.FileOutputStream.write(FileOutputStream.java:326) ~[na:1.8.0_71]
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.8.0_71]
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.8.0_71]
        at java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[na:1.8.0_71]
        at org.zeroturnaround.exec.stream.StreamPumper.run(StreamPumper.java:137) ~[zt-exec-1.8.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_71]
2016-01-27 14:32:51,783 ERROR [main] [o.z.e.c.StandardProcessCloser] Failed to close process output stream:
java.io.IOException: Stream closed
        at java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) ~[na:1.8.0_71]
        at java.io.OutputStream.write(OutputStream.java:116) ~[na:1.8.0_71]
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.8.0_71]
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.8.0_71]
        at java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[na:1.8.0_71]
        at org.zeroturnaround.exec.close.StandardProcessCloser.closeStreams(StandardProcessCloser.java:53) [zt-exec-1.8.jar:na]
        at org.zeroturnaround.exec.close.StandardProcessCloser.close(StandardProcessCloser.java:43) [zt-exec-1.8.jar:na]
        at org.zeroturnaround.exec.WaitForProcess.call(WaitForProcess.java:118) [zt-exec-1.8.jar:na]
        at org.zeroturnaround.exec.ProcessExecutor.waitFor(ProcessExecutor.java:1007) [zt-exec-1.8.jar:na]
        at org.zeroturnaround.exec.ProcessExecutor.execute(ProcessExecutor.java:860) [zt-exec-1.8.jar:na]
        at TryIt.main(TryIt.java:8) [classes/:na]

This same error occurs for any program that doesn't read from stdin, such as /bin/false, /bin/ls, /bin/pwd, etc -- this seems to be the difference between success and failure (i.e. no errors when using /bin/cat for example).

This only happens on Java 1.8 -- testing on 1.7 works fine. I have only used zt-exec 1.8 during testing.

I'm having this same error using Apache commons-exec, which is why I started looking for alternatives and come across this library.

Code to reproduce:

import org.zeroturnaround.exec.ProcessExecutor;
import java.io.ByteArrayInputStream;

public class TryIt {
  public static void main(String[] args) throws Exception {
    ByteArrayInputStream bais = new ByteArrayInputStream("foo".getBytes());
    ProcessExecutor exec = new ProcessExecutor().command("/bin/true");
    int exit = exec.redirectInput(bais).readOutput(true).execute().getExitValue();
    System.out.println(exit);
  }
}

Java installation:

openjdk version "1.8.0_71"
OpenJDK Runtime Environment (build 1.8.0_71-b15)
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)

Waiting for stdout, stderr pumping threads forever

Hello,

I've tried to spawn sh -c 'sleep 60' and then to kill this child process with kill -9 from the shell.
Although WaitForProcess successfully determines that child process died, PumpStreamHandler is waiting for stdout, stderr pumping threads to finish, that may never happen due to the following issue: Process.getInputStream().read() method hangs for destroyed Process.

Is it possible to add an option to this lib, that will make PumpStreamHandler to wait for the threads to die in the timely manner? This option will be pretty handy for cases as completing the parent process when the child one dies unexpectedly.

Kind Regards,
Sergey

Improved logic w/ input vs. output streams

For various reasons, I need to supply my own wrapped System.in inputstream. The problem I'm running into is that the ExecuteStreamHandler makes a very important decision on what kind of InputStream pump it creates IF the inputstream is System.in. In my case, the process runs and exits, but the InputStream is hanging forever on a blocking read() call.

I'm not sure this is the right logic for input streams vs. output. In fact, if the process actually does exit, why aren't all the streams closed and their pumping threads interrupted? Or if the output streams are closed, it seems like the inputstream w/o a doubt should be closed.

Running rpmbuild with --define '_topdir /path/to/my/rpmbuild/' does not work

I have been trying to execute a run of rpmbuild while setting av special --define config on the commandline. This does not seem to work with the ProcessExecutor. Running the same command ProcessBuilder works fine.

The following code:

public static void main(String[] args) {

    String[] command = new String[3];

    command[0] = "/usr/bin/rpmbuild";
    command[1] = "--define";
    command[2] = "'_topdir /path/to/my/rpmbuild/'";

    // ProcessBuilder
    try {
        ProcessBuilder pb = new ProcessBuilder(command);
        Process process = pb.start();

        InputStream is = process.getInputStream();
        InputStreamReader isr = new InputStreamReader(is);
        BufferedReader br = new BufferedReader(isr);
        String line;
        System.out.printf("Output of running %s is:\n", Arrays.toString(command));
        while ((line = br.readLine()) != null) {
            System.out.println(line);
        }

        int exitValue = process.waitFor();
        System.out.println("\n\nExit Value is " + exitValue);
    } catch (InterruptedException e) {
        e.printStackTrace();
    } catch (IOException e) {
        throw new RuntimeException(e);
    }

    // zt-exec
    try {
        ByteArrayOutputStream output = new ByteArrayOutputStream();

        new ProcessExecutor()
                .command(command)
                .readOutput(true)
                .redirectOutput(output)
                .execute();
        System.out.println(output);
    } catch (IOException | InterruptedException | TimeoutException e) {
        throw new RuntimeException(e);
    }
}

produces the following output:
Output of running [/usr/bin/rpmbuild, --define, '_topdir /path/to/my/rpmbuild/'] is:

Exit Value is 0
15:38:26.700 [main] DEBUG o.z.exec.ProcessExecutor - Executing [/usr/bin/rpmbuild, --define, '_topdir /path/to/my/rpmbuild/']...
15:38:26.704 [main] DEBUG o.z.exec.ProcessExecutor - Started java.lang.UNIXProcess@4e6e076c
Disconnected from the target VM, address: '127.0.0.1:54434', transport: 'socket'
15:38:26.708 [main] DEBUG o.zeroturnaround.exec.WaitForProcess - java.lang.UNIXProcess@4e6e076c stopped with exit code 0
error: Macro % has illegal name (%define)
error: Macro % has illegal name (%define)
error: Macro % has illegal name (%define)

Meaning that the first run produces no error from the rpmbuild, while the zt-exec based run does not set the args correctly, making the rpmbuild produce errors.

Am I doing something wrong, or is the ProcessExecutor not not handling commands with '

Mid-execution processing of streams

I can't seem to find a straightforward way to create or pass in some threads to consume stderr/stdout streams while a process is executing. Simple use cases for this are updating progress information based on console output of the process, etc.

If this is something reasonably easy to do (e.g. implement an interface, pass in a Runnable/Callable, etc.) then documentation for that would be helpful.

Thanks!

LogOutputStream works different than ProcessOutput regarding line breaks

The ProcessOutput produces different lines than the LogOutputStream for the same given string. The line breaks are treated differently. Especially if the input only uses CR as line break the LogOutputStream only detects the first CR as line break and does not create lines for subsequent CR.

project doesn't compile with java 8

Due to the new stricter linter in java 8 for the JavaDoc the project doesn't compile right now. There are about 60 errors of bad JavaDoc.

Support kotlin continuations

e.g. feel free to adapt if useful, I'm using this in my own project.

import okio.ByteString
import org.apache.commons.io.input.NullInputStream
import org.zeroturnaround.exec.ProcessExecutor
import org.zeroturnaround.exec.ProcessResult
import org.zeroturnaround.exec.listener.ProcessListener
import org.zeroturnaround.exec.stream.slf4j.Slf4jStream
import java.nio.charset.StandardCharsets
import kotlin.coroutines.experimental.suspendCoroutine

data class ExecResult(val exitCode: Int, val output: ByteString?) {
  val success = exitCode == 0
  val outputString by lazy {
    output?.string(StandardCharsets.UTF_8)!!
  }
}

val stdErrLogging = Slf4jStream.ofCaller().asInfo()

suspend fun exec(vararg command: String): ExecResult {
  return exec(command.toList()) {
    readOutput(true)
    redirectError(stdErrLogging)
    redirectInput(NullInputStream(0))
  }
}

suspend fun exec(command: List<String>, configure: ProcessExecutor.() -> Unit = {}): ExecResult {
  var outputRequested = false

  val pe = object : ProcessExecutor() {
    override fun readOutput(readOutput: Boolean): ProcessExecutor {
      outputRequested = readOutput
      return super.readOutput(readOutput)
    }
  }

  pe.command(command.toList())

  configure(pe)

  return suspendCoroutine { cont ->
    pe.addListener(object : ProcessListener() {
      override fun afterFinish(process: Process, result: ProcessResult) {
        try {
          val outputString = if (outputRequested) {
            val output = result.output()
            ByteString.of(output, 0, output.size)
          } else {
            null
          }

          cont.resume(ExecResult(result.exitValue, outputString))
        } catch (e: Exception) {
          cont.resumeWithException(e)
        }
      }
    })

    try {
      pe.start()
    } catch (e: Exception) {
      cont.resumeWithException(e)
    }
  }
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.