Question: RedirectOutput(CachingOutputStream) about zt-exec HOT 15 CLOSED

Hi,
using the LogOutputStream already buffers the lines that you read. If you can't read them quickly enough it means the executed process can't produce more output and the corresponding thread blocks until you have read more output.
By default stdout and stderr are merged together. In case you have changed that your LogOutputStream would miss all the lines from stderr.
My question would be how do you know it's missing some lines? Maybe the process just doesn't print those lines.

from zt-exec.

I do it the following (leave it merged):

    public void executeInBackground(IPluginExtension IPluginExtension){
        try {
            reConfigure();
            StartedProcess startedProcess = this.redirectOutput(IPluginExtension.getResultProcessor()).start();
            future = startedProcess.getFuture();
            process = startedProcess.getProcess();

        } catch (IOException e) {
            e.printStackTrace();
        }
    }

IPluginExtension extends LogOutputStream and analyzes the buffered line in processLine().
In processLine() I print the line in stdout for debug purpose and there I saw when I remove the string analysis it prints 2 more lines from my command line process.

from zt-exec.

Maybe it's the 2 more lines in the end. Your additional code may throw an exception so it stops the thread that's reading the output. Add try-catch to your code to see whether that's the case.

from zt-exec.

Okay thanks, no I can see my mistake -.-

from zt-exec.

You're welcome.

from zt-exec.

Hello,

I have a new problem with ZT-Exec and can't hunt it down (try n catch doesn't show the problem either)

        @Override
        protected void processLine(String line) {
            try{
                System.out.println(">>> " + getPluginId().toUpperCase() + ": " + line);
            }
            catch (Exception e){
                e.printStackTrace();
            }
        }

Same configuration like above:

• Windows 7, java 1.8.0_45
• Commandline tool John the Ripper, gets used as following:
  "C:\john\john\run\john" --wordlist="C:\john\john\run\realhuman_phill.txt" --format=raw-md5 --rules "C:\john\temp-hashlist.txt"

The process is running and can be destroyed as well, but it seems that the stdout does not get captured anymore after a few lines...
Running the command line tool in cmd (Windows 7) manually works without problems and shows the missing lines...

Is it a problem, when the command line tool uses multi-threading?

Do you have any idea how to debug this problem?

from zt-exec.

Hi,
how did you exactly execute the command from command line?

1. Some apps may distinguish whether they are run with a console or not, e.g. in Java there's also java.io.Console.
2. Some apps may read from stdin, if it's open wait for data, if it's closed continue with another logic.
   By default ProcessExecutor closes the stdin of the forked process just after it's started.

To investigate the issue I would try it with plain ProcessBuilder. Since Java 7 you can just call ProcessBuilder.redirectOutput(Redirect.INHERIT) to pass the forked process' stdout to your own stdout directly. It uses file handles directly instead of pumping.
Let me know how it goes.

from zt-exec.

Thank you very much for your help!

Like I wrote above, I execute my command in a subclass of ProcessExecutor.
(and print the stdout and stderr lines later in some IPluginExtension to System.out)
The command for the ProcessExecutor looks like this:

"C:\john\john\run\john" --wordlist="C:\john\john\run\realhuman_phill.txt" --format=raw-md5 --rules "C:\john\temp-hashlist.txt"

The same command (copy and paste) I use in the console manually and get more stdout results...
I also tried adding "cmd \c" before...without difference..

1. Since my command line tool is running and working at the beginning and with a smaller wordlist for john...I don't think it checks whether it is executed in a console or not (otherwise it wouldn't start in the first place?)
2. I tested the command line tool manually in the console and it does not require any user inputs

I tried the following test:

        try {
            processBuilder = new ProcessBuilder(commandString).inheritIO().redirectOutput(ProcessBuilder.Redirect.INHERIT);
            startedProcess = processBuilder.start();

            InputStream is = startedProcess.getInputStream();
            InputStreamReader isr = new InputStreamReader(is);
            BufferedReader br = new BufferedReader(isr);
            String line;
            while ((line = br.readLine()) != null) {
                System.out.println(line);
            }

        } catch (final Exception ex) {
            ex.printStackTrace();
            System.exit(1);
        }

And it is the same result as with ProcessExecutor, the command line output gets not captured anymore after the first 5 lines...

To compare:

"C:\john\john\run\john" --wordlist="C:\john\john\run\password.lst" --format=raw-md5 --rules "C:\john\temp-hashlist.txt"

The command with a small wordlist runs without problems and everything gets captured from ProcessExecutor.

"C:\john\john\run\john" --wordlist="C:\john\john\run\realhuman_phill.txt" --format=raw-md5 --rules "C:\john\temp-hashlist.txt"

The command with a big wordlist, that takes a long time, doesn't get captured...
Results from john, get written in stdout and also for session resume purpose in a file.
Executing the command with long wordlist in console writes the results in a file...and runs quite a while....
Executing the command with long wordlist in ProcessExecutor results in a running john process, that prints the first 5 lines in stdout and the rest not and also not in the external file....

Could it be that John might start subprocesses/threads to process the big wordlist and those outputs dont get captured by ProccessExecutor?

from zt-exec.

1. As ProcessExecutor as ProcessBuilder acted the same way it's not an issue with ProcessExecutor.
2. In case the output wasn't appended to an external file I'm quite sure John didn't print the output at all. So it's not an issue with reading the output in Java but it might depend on how Java forks system processes.

Standard streams belong to each process so multiple threads don't change a lot here. If John uses sub processes it would inherit their IO regardless John is started via console or not. So it shouldn't change much either.

You could try switching Jonh's version, JVM version (32-bit vs 64-bit, also the version) and also OS (use Linux or Mac OS X instead of Windows).

from zt-exec.

I tried following configurations:

Initial configuration was:
Win7 64Bit, Java 1.8.0_45 64Bit, ProcessExecutor, John 1.7.9-jumbo-5 for win

• Win7 32Bit, Java 1.8.0_45 32Bit, ProcessExecutor, John 1.7.9-jumbo-5 for win
  --> same result
• Linux Mint 17 64Bit, Java 1.8.0_60 64Bit, ProcessExecutor, john-1.7.9-Linux-x86-64
  --> Had to change some slashes and " and then the example with small wordlist worked, with big wordlist I noticed something interesting:
  First of all the first 5 lines were not coming (because in windows they only said some warnings about windows paths...)...but I expected this, since john is originally a linux tool.....
  Then...no output like in the windows version, but when I stopped the program (over my IDE's stop button) it showed the abortion output of john and also the desired output showed up in johns session file.

But I still don't know the reason for the problem now...
Could it be that the java vm heap size has to be increased or some other additional java vm argument? (already tested to increase -Xms512m and -Xmx1024m without success)

from zt-exec.

By no output did you mean it stopped without output or it hang without output.
In the latter case it might just run long and buffer the output. In this case terminating caused it to flush before exiting. If John isn't a Java tool -Xms shouldn't affect it at all.

from zt-exec.

It hang and the output got flushed when the process got killed, could be a reason yes.
But why does the output gets flushed earlier when executed manually in console?

I think about three more options to test:

• piping the output in a file and read the output then continuously out of the file
• executing a shell script, that executes john
• switch to python...

I was just hoping that a library could abstract from all those well known, old errors....

from zt-exec.

John seems to buffer its output as mentioned here: http://www.openwall.com/lists/john-users/2006/03/19/3
However it may detect a console and use line-buffering in that case. If you started an intermediate shell script from JVM it still wouldn't have a console I think. In this case python wouldn't work either but you may try it to prove if that's the case.
Maybe the --show option helps?
If it flushes more often to a file you may direct the output to a file and "tail -f" it from there. Here's how to do it from Java: http://stackoverflow.com/questions/557844/java-io-implementation-of-unix-linux-tail-f

from zt-exec.

Just a short answer what I did in the meantime...

I tried to execute it over a bat, piping the output in a file..to read it later with tail:

cmd /c start john-plugin.bat

In john-plugin.bat is then:

"C:\john\john\run\john" --wordlist="C:\john\john\run\password.lst" --format=raw-md5 --rules "C:\john\temp-hashlist.txt" > tmp.txt

Same behavior...john thinks its not executed in a console and buffers everything until process exit.

Found a similar question...but with no satisfying answer:
http://stackoverflow.com/questions/21003632/no-input-from-processbuilders-inputstream-until-external-process-is-closed

stdbuf for linux would be an option...but I need os-independence.

Okay next option would be to use --show in parallel when john is running...

from zt-exec.

This proves that the issue is caused by John itself. Let's hope the --show option can configure it properly.

from zt-exec.