muter-mutation-testing / muter Goto Github PK
View Code? Open in Web Editor NEWπ Automated mutation testing for Swift π³οΈ
License: MIT License
π Automated mutation testing for Swift π³οΈ
License: MIT License
This is part of #54.
Makes it more clear what it does. Also matches naming from .swiftlint.yml.
I think there are some subtle discrepancies between how muter refers to things and how the wider community refers to certain concepts and mutation testing. Go through and make sure it's consistent.
Right now, Muter doesn't anchor it search for a file name to the very end of a string, end instead simply does a contains
check. This causes files like .swiftlint.yml
to be discovered by Muter as a file that could potentially contain mutable source code.
This story would update Muter so it starts looking for a file extension at the very end of a file path or name. This would remove burden from a user who would currently need to explicitly ignore things like a .swiftversion
or .swiftlint.yml
file
Right now, Muter has a limitation that requires there to be spaces around operators so it can mutate them. The story is supposed to add a step that eliminates this limitation.
We can probably use swiftlint to do some very basic formatting of code Muter is operating on after it copies it to a temporary directory, but before it begins doing any file or mutation operator discovery.
This is part of #46
If a user misconfigures Muter, and that misconfiguration prevents their test suite from being run (e.g because xcodebuild
is unable to locate a scheme), Muter will incorrectly report that is a successful mutation test. This issue is to improve Muter's detection of tooling errors, and to prevent them from becoming successful mutation test runs.
Falling out of a conversation and some research done on #3, it makes sense to have Muter copy the code base it's testing into a temporary directory - /tmp
is most likely.
Muter will still need to restore files after mutating them, but prior to running the test suite again, so the existing code will probably stay the same. This issue adds a step prior to source file discovery or mutation discovery which moves the entire code base that will be analyzed into a new directory. Additionally, there's the potential that there will need to be some (likely very) small rework in associating this new directory with the existing code that uses file paths.
It's important to know if a test suite is aware and testing all of the side effects that are in a code base. Muter should have a way of detecting when a line of code is intended to cause a side effect, and have a way of causing the code to not cause that side effect (probably by deleting the line).
Right now, Muter produces a lot of text as a result of running xctest
so much. To make the output more approachable and easy to digest, this output should be suppressed somehow, potentially written to disk so it can be reviewed later to diagnose issues.
Right now, the only thing getting in the way of this is using macOS specific FileManager functions. If we moved away from that, that it should just work on linux
When running Muter on BonMot, I observed this behavior across several applied mutations.
StylisticAlternates.swift Line: 204, Column: 41 Side Effects passed
Tracking.swift Line: 23, Column: 21 Negate Conditionals passed
StyleableUIElement.swift Line: 84, Column: 38 Negate Conditionals passed
It's possible that this can be fixed by #8. However, this speaks to the need to improve how Muter determines whether or not a test suite is failing.
Interestingly enough, this behavior is inconsistent - there are times when Muter accurately determines that a test suite is failing or passing. Perhaps this bug is caused by an overreliance on the status code returned by xcodebuild
.
Or it could even be Swift? JSON isn't well suited to things like having list of command line arguments. YAML is nice because it has bare words, so you don't have quote marks mucking up the works.
Ejecting out of xcode broke the CLITests.swift
code. These are valuable acceptance tests, and the only way that main.swift
gets exercised in Muter's test suite. Fix this!
Right now, if you interrupt Muter while it's running, it's most likely that a user's codebase will still contain a mutant. This issue is to improve Muter's cleanup process by registering handlers for UNIX signals.
Right now, Muter will mutate operator declarations. This is an issue because it causes a compilation failure, which is needless and only slows Muter down.
Take this code as an example:
extension AbsolutePosition: Equatable {
public static func == (lhs: AbsolutePosition, rhs: AbsolutePosition) -> Bool {
return (lhs.column, lhs.line, lhs.utf8Offset) == (rhs.column, rhs.line, rhs.utf8Offset)
}
}
This is an operator that is necessary for AbsolutePosition to be equatable; mutating it is only going to break the code, which isn't a very meaningful test failure to report on, especially given how long Muter takes to run
see how stryker defines it for an example: https://github.com/stryker-mutator/stryker-handbook/blob/master/mutator-types.md#assignment-expression
Presently, Muter doesn't explicitly test for this case but should. This story is really just about adding a test for these cases with the Remove Side Effect Operator.
Create a regression test suite which determines if the number of mutation operators, or the reported mutation scores, change unexpectedly when a new commit is added to Muter.
A main problem any tool that modifies source code has is knowing whether or not source code changes are going to cause static or dynamic errors within the program.
I have a hypothesis that modifying conditional logic inside a guard statement will nearly always lead to a static or dynamic error. This is partially proven by running Muter on a few code bases and noticing this tendency.
It could be valuable to have Muter avoid mutating logic that's inside a guard statement. This would speed up Muter by not having it make source code changes that would only lead to a compilation or fatal error.
Right now, the logs produced by Muter are pretty noisy. Scattered amongst all of the output of xcodebuild
are the various actions that Muter performs. There's probably a way by using FileHandle
to improve the legibility of the output while not losing the constant feedback which let's the user know that Muter is actually running.
It should populate it with reasonable defaults or, for bonus points, ask you for input and then build it. At the very least, the error message when the config file is missing should include a URL to the config section in the readme.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.