stryker-mutator / stryker-js Goto Github PK

We should not allow for implicit any type in TypeScript. Variables without type annotations are usually a mistake. After all type info is added we should enforce that we don't get implicit any-typed variables by setting noImplicitAny: true in tsconfig.json.

We work on the typescript-migration branch. Please keep this todo list in sync:

• mutations/BaseMutation.ts
• mutations/BaseOperatorMutation.ts
• mutations/ConditionalBoundayMutation.ts
• mutations/MathMutation.ts
• mutations/RemoveConditionalsMutation.ts
• mutations/ReverseConditionalMutation.ts
• mutations/UnaryOperatorMutation.ts
• reporters/BaseReporter.ts
• reporters/ConsoleReporter.ts
• utils/FileUtils.ts
• utils/ParserUtils.ts
• utils/TypeUtils.ts
• testrunners/BaseTestRunner.ts
• testrunners/JasmineTestRunner.ts
• testrunners/KarmaServerWorker.ts
• testrunners/KarmaTestRunner.ts
• AbstractSyntaxTreeNode.ts
• Mutant.ts
• MutationRegistry.ts
• Mutator.ts
• ReporterFactory.ts
• Stryker.ts
• TestFile.ts
• TestResult.ts
• TestRunnerFactory.ts

Tests:

• test/integration/testrunners/JasmineTestRunnerSpec.ts
• test/unit/MutantSpec.ts
• test/unit/MutationRegistrySpec.ts
• test/unit/MutatorSpec.ts
• test/unit/ReporterFactorySpec.ts
• test/unit/StrykerSpec.ts
• test/unit/TestFileSpec.ts
• test/unit/TestResultSpec.ts
• test/unit/TestRunnerFactorySpec.ts
• test/unit/mutations/BaseOperatorMutationSpec.ts
• test/unit/mutations/RemoveConditionalsMutationSpec.ts
• test/unit/reporters/ConsoleReporterSpec.ts
• test/unit/testrunners/BaseTestRunnerSpec.ts
• test/unit/testrunners/JasmineTestRunnerSpec.ts
• test/unit/testrunners/KarmaTestRunnerSpec.ts
• test/unit/utils/FileUtilsSpec.ts
• test/unit/utils/ParserUtilsSpec.ts
• test/unit/utils/TypeUtilsSpec.ts

For every Jasmine test that is ran, a new nodeJS process, a Karma server and a PhantomJS is started. This wastes a lot of time. By keeping the nodeJS process, the server and the browser running in between tests, tests could be ran much faster.

A proof of concept for this feature has been made by Nico which shows it is possible to keep the processes running while collecting code coverage for every test. The proof of concept can be found here.

Support has to be added for the QUnit test framework.

We want a more clean the mutation api. It should be as easy as possible to create mutations.

Right now, most promise reject's are silently ignored. Before we release v1, we should add error handling everywhere.

We should rewrite the karma test runner to keep the karma server open instead for the entire test run, instead of opening up a new node process for each mutation test run. This can be done using the way of working in the karma-coverage-per-test prototype.

We can maybe also add a new abstraction for starting child processes and handling them, so we don't need to recreate it at a later stage (server side unit tests).

We should move styker-api, stryker-karma-runner, stryker-jasmine to seperate repositories before release 1

We should add dev-dependencies to them to add integration tests.

As we moved to typescript, it would be relatively easy to move to attributes instead of getters-setters. If we need logic in the future, we'll replace the attribute with an ES 5 property.

Server sided Mocha tests are not yet supported by stryker. See #11 for browser based Mocha test support.

Right now: we always load the source files before other files. This approach will not work when you need to load library files.

For example: you need to load angular, after that your own source files, after that your test files. Karma solves this by having one big list of files. Thus you have fine grained control of the order of files.

I'm thinking of introducing the same principle for stryker. Instead of source files and other files, you'll have files and source files. The files list also includes the source files and gives you fine grained control of the order of files. The source files are the files we are going to mutate.

If the source files contains a file which is not included in the files list, we'll either have to give an error, or load the files before or after the 'normal' files.

@simondel what do you think?

If Stryker was unable to mutate the source code, the rest of the mutation testing process still continues (starting multiple test runners, logging the mutation score etc), while we can simply quit if there are no mutants.

Browser based Mocha tests are not yet supported. This feature could use the Karma test runner. See #10 for server sided Mocha support

As we moved to typescript classes, it is easy to recognize private variables. We don't need the underscore _ prefix anymore.

For example:

node dist/src/Stryker.js  -m test/sampleProject/src/*.js -f test/sampleProject/src/*.js,test/sampleProject/test/AddSpec.js,test/sampleProject/test/CircleSpec.js

Expected:

ERROR: One or more tests failed in the inial test run:
ReferenceError: Can't find variable: someGlobalVariableThatIsNotDeclared
at c:/z/github/infosupport/stryker/test/sampleProject/src/Error.js:9 

Actual:

ERROR: One or more tests failed in the inial test run:

It should be possible to configure which mutation types are and which aren't used. The importance of this feature increases as the number of mutation types increases.

I would like to add a way to read config using a config file instead of command line arguments. Very similar to what karma uses.

Note: the current command line arguments would NOT be replaced with a config file

Proposal:

• When you call stryker you can supply a config file with --configFile (or -c?). If you don't supply one it will use stryker.conf.js as a default.
• the config file should look like this:

module.exports = function(config) {
    config.set({
      files: [
           'lib/**/*.js',
           'src/**/*.js',
           'test/**/*.js'
       ],
       mutate: [
            'src/**/*.js'
       ],

        // additional optional options with default values: 
         testFramework: 'jasmine',
         testRunner: 'karma',
         timeoutMs: 2000,
         timeoutFactor: 1.5,
         plugins: ['stryker-*'],
         port: 9234
    });
};

• By default: any plugin is loaded from the node_modules folder starting with the name stryker-.
• A plugin will also have the possibility to register a ConfigWriter to the ConfigWriterFactory.
  • Config writers will be applied after the reading of the normal stryker.conf.js file. Indeed it has to, because that file defines the plugins to be loaded in the first place.
  • For example: a plugin with the name stryker-karma-runner could register an additional ConfigWriter with the name 'karma' to the ConfigWriterFactory. That KarmaConfigWriter can look for a property with the name karmaConfigFile in the existing config. It can read that karma.conf.js file and than choose to override some of the settings in the stryker.conf.js file. For example override the files based on the karma.conf.js's files key.
  • This way, plugin implementers can have maximum flexibility in adding config based on config settings that live outside the stryker process.

Does everybody agree with this proposal?

• Note: I just changed some parts. This will not replace the command line arguments, just add a new one. I would like to maybe add a stryker-cli in the future where you would only accept a config file.*

See #29 , we should create the jasmine test framework in order to pick and choose tests.

TypeUtils was designed to add runtime checks to validate the types. We don't need it anymore as we moved to compile time checks with typescript.

We should use underscores in all module names

See if we need better option names. Discussion points:

• testFiles+libs ==> otherFiles/files/additionalFiles

I discussed this feature with colleague @Jaapapa.

Basically we would like a sonar qube plugin. This plugin would output a single metric (at first): the mutation score. A sonar qube adminstrator would then be able to add a threshold to that metric, so when the mutation score falls below 80%: your sonar qube quality report is red.

To do this we should define these steps:

1. Create a json/xml reporter. This reporter does not have to be sonar specific, so it would be great to make it generic.
2. Create a Sonar Qube plugin. This will be written in Java, so should be its own repo. Maybe we can look to sonar-pitest, only that reports issues instead of a metric. According to @Jaapapa, we should be able to add a custom metric/measure for mutation score. This plugin will work with the default sonar qube runner executable or with the sonar qube maven plugin.
3. Add docs which explain how to configure the quality gate and how to use the plugin.

The Typescript files should be compiled to JavaScript files during a NPM install

It seems that Karma-coverage is reporting more than expected. For example: the result of Add should be able to add two numbers (see below).

I will bring forward the in-memory karma-coverage reporting to see if that solves this issue

EDIT Removed too long example:

(User created) properties of objects could be removed to mutate code.

Right now, all code is being mutated. Code coverage has been collected before code is mutated. This makes it possible to only mutate code that has actually been tested. It is possible that only mutating tested code is slower than mutating everything due to the fact that code coverage has to be analysed.

This feature should not replace the current 'mutate everything' system. It should be an option (or the default, making 'mutate everything' an option).

In certain situations, the mutation testing process can get stuck. While this happens, several nodeJS, Karma and PhantomJS processes are still running.

If stryker freezes in a certain project, it will always freeze at that point. However, if another pc is used it can freeze at another point in the mutation run, or it does not freeze at all. In some cases stryker only freezes when the option individualTests is turned on.

The issue can be fixed by enabling logging of nodeJS, Karma and PhantomJS processes. A temporary workaround has been added which reroutes the stdout of the child processes to an empty function. This enables the logging (which fixes the freezing) while not actually showing anything.

We need a new api for Mutant. Right now: mutants handle their own file IO. The save the mutation to disk to a file of their choosing (using fileUtils.createFileInTempFolder). There are 2 issues with this way of working.

1. we want to spin up an n number of test runners. After a test runner is created, we cannot change the targeted files by that test runner (Karma doesn't support it). The only way to do this (that i can think of) is that we copy over all source code to a separate folder before hand. That copy of the source files will be created per test runner and given to the test runner at construction time. A mutation can than be made in that copy of the source files. These can also run in pure isolation, as long as every mutation gets reverted after wards
2. All IO is done synchronously. This is unnecessary, because we should be able to run multiple mutations at once on multiple test runners. It would be a shame if all testrunners are waiting on a blocking IO call

I would suggest the following API per test mutation

Mutant
+ sourceFile: string; // path to original source file`
+ mutationPosition: Location; // location of the line to be mutated
+ substitude: string; // the piece of code to substituted (the actual 'mutation)

The orchestration of which source file corresponds to which source file per test runner should be done outside of the Mutant class

@simondel Do you agree?

It would be nice to have support for multiple severities for the console reporter such as info or debug. This gives the user the option to see as much or as little output as he/she wants.

We need a facade to start a test-runner in a child process. This way, every test runner can stay clear of starting child processes itself. The timeout can also be much easier implemented in this facade, because he can simply kill the child process if it takes too long (and start a new one).

This can be easily implemented using an adapter pattern.

Can we get a contributor guide in the repo so people know how they can get their changes included?
I have a standard template ready to go: https://github.com/infosupport/contribute-md-template

If you like I can make a pull request for it as well :-)

As we want to transform our code to TypeScript, the first step will be to transform our code *.ts files. We should add those to source control instead of our js files. We should also add a compile step into our build system

Entire statements could be removed to mutate code

If a test fails during the initial test run, the text: ERROR: One or more tests failed in the inial test run! is logged. A user won't know which test failed, making it difficult to find the issue.

If a test fails during the initial test run, the user should see which test(s) caused the error.

Tsd is depricated, we should move to typings for our typescript typings.

When Karma errors in the initial test run it is treated as if there are no tests. When you turn off the silent flag you can see the following output:

INFO: Running initial test run
using config {"browsers":["PhantomJS"],"frameworks":["jasmine"],"autoWatch":false,"singleRun":false,"plugins":["karma-jasmine","karma-phantomjs-launcher","karma-coverage"],"files":[...],"port":1234,"coverageReporter":{"type":"in-memory"},"preprocessors":{...},"reporters":["coverage"]}
24 03 2016 22:14:08.982:INFO [karma]: Karma v0.13.22 server started at http://localhost:1234/
24 03 2016 22:14:08.986:INFO [launcher]: Starting browser PhantomJS
24 03 2016 22:14:09.494:INFO [PhantomJS 1.9.8 (Windows 8 0.0.0)]: Connected on socket /#tOMWTOX3l_ZzzTYYAAAA with id 67783735
karma run done with  1

I've tested this using our internal KarmaDesigner project with the following arguments:

    "program": "${workspaceRoot}/node_modules/stryker/dist/src/Stryker.js",
            "stopOnEntry": false,
            "args": ["-s","app/scripts/app.js,app/scripts/httpConfig.js,app/scripts/resources/ConditionRsc.js,app/scripts/resources/PageDtoRsc.js,app/scripts/resources/SurveyResultRsc.js,app/scripts/resources/SurveyRsc.js,app/scripts/resources/SurveySummariesPageRsc.js,app/scripts/routes.js,app/scripts/services/dragDropHelper.js,app/scripts/services/idGenerator.js,app/scripts/services/jsonHelper.js,app/scripts/services/localStorage.js,app/scripts/services/modalHelper.js,app/scripts/services/notificationStorage.js,app/scripts/services/random.js,app/scripts/services/surveyFind.js,app/scripts/services/surveyForEach.js,app/scripts/services/surveyStorage.js,app/scripts/services/tenantHelper.js","-o","test/spec/controllers/editFactRuleSet.test.js"],

For clarification: i deliberately not included the angular js sources in the otherFiles list. I would expect Stryker to show me this (cannot call module on undefined or something)

All map types in test_runner/Coverage.ts are typed using a number indexer. For example:

export interface StatementMap {
  [ref: number]: Location;
}

Although in practice, the ref is always a number, we should change it to be string. This because the usage is somewhat strange:

coveredFile.statementMap[parseInt(statementId)]

With parseInt we convert the string to a number, but for the indexer [] it is converted back to a string. This is kind of pointless.

The base timeout is not saved with the properties of a test when it is placed in a queue. This timeout is read from the config when the test is ran. In many cases, the timeout has already been overwritten by the base timeout of another test.

Projects which include HTML files (such as AngularJS projects) are not properly supported. Support for these projects has to be added. This feature could be tested on an AngularJS project to ensure it works.

Certain array elements could be removed as part of a mutation.

Integrate the new TestRunner and TestFramework apis in Stryker. Make sure to support the options we already had, maybe rename them.

For the TypeScript migration we should replace all constructor functions by actual typescript classes. We'll start doing this in the typescript-migration branch and comment here when we convert each file.

We should create the Stryker 'api' in a seperate folder. In this folder we should create the new interface definitions for TestFramework and TestRunner.

We should describe API as solution 2 on the image below.

escodegen applies formatting to the code. This usually doesn't matter, but in a few cases it does. This causes the mutated line of code to be shown incorrectly because line numbers have changed.

An example can be seen using the escodegen online demo by using the following code:

var arr = ['A', 'B'];
var obj = { name: 'Simon', age: 22 };

A possible solution is to ignore this feature of escodegen and search the AST for the code which should be on the specified line number. By generating this part of the AST and removing all line breaks, the correct line of code could be shown in the report.

The stryker mutation test process seems to hang for a few seconds before existing. This is due to the timeout set in the IsolatedTestRunnerAdapter (it is never canceled)

HTML reporting can make the final report of the mutation test more readable. This HTML report can be similar to a HTML code coverage report.