Identified issues about muter HOT 7 OPEN

Hi @felipecp! I'll try to answer your questions

High disk usage
This happens because we basically need to create copies of file for each mutant. To fix that we need to discover
mutants dynamically with subsumption relations, which we, unfortunately don't have at the moment.

Time to run is too long
Mutation tests are known for being slow. One thing you could do is filter (using --files-to-mutate or skip branches of code.
This PR can also improve the overall run time.

The compatibility is not good depending on Xcode version
It happens because we use a weak link to the SwiftSyntax dylib, that is bundle within Xcode. If that changes, it will break muter.

from muter.

We could probably delete copies of the project after we're done with them, right? I forget whether each mutation is a full new copy of the project, or whether it's more efficient than that. Also, note that APFS means that you're likely not actually incurring all that disk space, since you're only paying the cost of new files for ones that you duplicate. (For a large project with many small files, the cost of the directory structure is probably nontrivial…but probably not 200GB either.)

from muter.

I need to double check, but I think we apply the mutation, run the tests and undo it.

@felipecp do you happen to know if that's on the temp folder? Or maybe it's the logs that are generated on each mutation round?

from muter.

Hi there, I'm with @felipecp.
In fact they are in a temp folder, but I cannot find out why muter (or xcode) is not deleting them.

The folders that occupied these 200gb are inside this path: /var/folders/k2

Muter found only 366 mutants but I see about 13.000 folders like:

(...)
com.apple.CoreSimulator.SimDevice.C41A0642-BD5D-435A-96B5-7E4A5D60DCE9.Standalone.FFFFFFFFF-2C5B-46DD-9D04-B7642C705017
com.apple.CoreSimulator.SimDevice.C41A0642-BD5D-435A-96B5-7E4A5D60DCE9.Standalone.FFFFFFFFF-3265-40BC-9A40-0001643B451E
com.apple.CoreSimulator.SimDevice.C41A0642-BD5D-435A-96B5-7E4A5D60DCE9.Standalone.FFFFFFFFF-A582-4633-9E1A-AA2DC73371A0
com.apple.CoreSimulator.SimDevice.C41A0642-BD5D-435A-96B5-7E4A5D60DCE9.Standalone.FFFFFFFFF-B6DD-4A6B-BC8A-EC74AA88544F
(...)

I'm not sure if these are only outputs of compilation but wouldn't you expect xcode to clean them? or should muter code call for this clean manually?
(and btw, just so you can estimate the size of the project to see if these 200GB makes any sense... I'm already ignoring some parts of the source code and yet it took muter 4h27min to run all mutations)

from muter.

I believe those folders are cleaned up by the OS on reboot, but muter should totally do a better job of cleaning up after itself. It could even potentially do it in the background so it doesn't interfere with the currently running tests.

Alternatively, it could do all mutations and reversions in-place instead of making a new copy for each change. I feel like it maybe even used to do this, and was removed for some reason? Or at least we had talked about it.

from muter.

Hi @ZevEisenberg and @rakaramos

Do you think I can create any kind of shell script that could delete these folders once they are done being used by muter?
The main problem I see is that I don't know how to make sure which ones xcode is still using and which ones can be deleted...

Ideas I have:

1. Each 10 minutes the service would delete the folders in this path older than 2 min
2. The service would somehow track xcode new builds (from logs or any other instruments tool) and delete olders once a new one is generated

Problems I see in these solutions:

1. Will this path be the same for all the runs of muter? Is 2 minutes really enough for muter to conclude it's work, I mean, how to make sure I'm not deleting files that are still being used?
2. No idea how to track this

from muter.

I think the right fix for this will be to update muter itself to delete old files, possibly with a flag to keep them around for debugging purposes. The folders are generated by the OS via a call to NSTemporaryDirectory() (I think), and it would be hard to know outside of muter which ones are no longer in use.

from muter.