Muter hangs when mutation change causes test hang about muter HOT 6 OPEN

Hello @aim2120 and thank you for reporting this 🙌

Have you tried executionTimeAllowance?

In addition, as a developer, I would like an opt-in way to force muter to exclude specific code from mutation. I imagine it like the following:

await withCheckedContinuation { continuation in
        fetchMessages { messages in
            // muter: skip
            continuation.resume(returning: messages)
        }
    }

from muter.

Hey @Nikoloutsos thanks for the response! Unfortunately executionTimeAllowance doesn't scale reasonably well for me. I'm working in a codebase with many other developers and trying to enable mutation testing for all of them, without needing to change all of their test code.

Honestly, the best solution I see is to allow muter to enable timeouts itself via config. That way muter can guarantee that it will complete, regardless of the behavior of the test executable.

I was trying some stuff out locally and it seems -test-timeouts-enabled and -maximum-test-execution-time-allowance aren't a great solution, since they automatically retry multiple times (I haven't found a way to disable this retry behavior using xcodebuild).

from muter.

hi @aim2120, could you provide a brief code snippet where I can reproduce the issue and check if the flags are enough?
In the mean time, you can disable the operators from being injected into this part of the code

from muter.

Sure, here's an SPM library that hangs when you run muter. This is because the mutation applied removes the line continuation.resume.

MuterHangingExample.zip

So there's actually a few negatives about the flags -test-timeouts-enabled and -maximum-test-execution-time-allowance that I didn't realize when I wrote the initial post:

• They only work if the test executable is xcodebuild, so we can't use with swift
• They will automatically retry the test, so that the test has to timeout multiple times before xcodebuild fails. This causes muter to take much much longer, since a timeout of 3 minutes for example becomes (3 * # of retries) minutes.

How would you feel about introducing timeouts to muter itself? This could be configured via the muter config, with a default value of nil if none is found, for backwards compatibility. This would mean that we can control the maximum test time, regardless of whether we're using xcodebuild or swift (or anything else), and doesn't require any fidgeting around with the limitations of the command executable.

from muter.

Oh, and just to reiterate what I said in my previous comment: I'm hoping to unlock using muter across many projects contributed to by multiple developers, so it's not feasible for me to figure out exactly which lines of code are potential hang points for mutations.

from muter.