Running external test processes about xunit-performance HOT 4 CLOSED

Yes, will be needed in Roslyn testing too.
I agree the test harness has to know about this difference and that should be expressed via the attributes. I think the current [Benchmark] attribute has too much implicit (hidden) cleverness. I would rather see some of that hoisted into a [SelfTuningBenchmark] (need a better name). The base [Benchmark] attribute should probably have nothing more than the iteration count and the (optional) warm-up flag. [ExternalBenchmark] would have include a time-out value, maybe an "expected exit code" (automatically fails the test) or an "expected output" (compare stdout/stderr to a string value). Perhaps an attribute isn't the right way to go about it; a static helper method would do just fine.

from xunit-performance.

If you guys come up with a better model then it would be interesting for us to adopt it. But in the mean time we had to get this done for our tests dotnet/wcf#43 and most likely will use this for our perf and stress.
/cc @roncain

from xunit-performance.

Yes, dotnet/wcf#43 represents our current strategy for being able to run external processes either locally or on another machine our tests need during their setup. The kinds of things we need these external processes to do may not be available in the framework where xunit is running (disqualifying xunit fixtures). The tests make http requests to a WebAPI app running independently, and that WebAPI app starts the appropriate process(es) with which the tests can interact. In our case, we are interested in the client-side (test-side) metrics, not the external process's.

from xunit-performance.

Now that iteration is under programmatic control in each test method, I'm not sure this topic is still relevant. Starting an external process per iteration is a simple matter of using Process.Start.

from xunit-performance.