Currently, shuffling perturbations independently permute instances for features being perturbed. This is ok for feature importance analysis, but for interactions analysis, the joint shuffling perturbations for an interaction pair should use the same permutations as used by its component features.

It was designed this way to allow a deterministic amount of noise to be added to corrupt the ground truth model, such that the noise varies across instances but is the same when the same instance is passed to model.predict. However numpy 1.17+ has slower RNG seeding, resulting in the tests timing out on Travis CI.

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

Using the -no-condor-cleanup flag should change the cleanup variable to False, rather than keeping it at the default True. Line 62 in master.py

Currently temporal features are required in the input datafile and if the perturbation mode is set to shuffling, perturb_temporal_data throws a ValueError halting the execution. For use cases where temporal features don't apply and/or one wants to shuffle instead of zeroing out, this causes an issue. It'd be great if temporal features are optional in the input file; and until shuffling temporal data is implemented, maybe throw an exception in perturb_temporal_data() at line 210 instead of an error and catch it in [perturb_features_for_record() near line 170] (

The code may error out when -analyze_interactions and -condor are enabled together.
To reproduce: Disable -cleanup flag in simulation.py; run 'pytest --basetemp=condor_test_outputs tests/condor_tests'
Cause:

• pipelines.py->SerialPipeline->features_filename assignment ignored by condor helper that writes features to file
• Condor helper compile_results picks up previously computed non-interaction results due to wildcard match

Fix:

• Change condor helper writer to accept input filename arg given serial mode
• Change condor helper result compiler to only process correct result file given serial mode

Use of model losses means that the null hypothesis may not be valid for non-interacting features, so -analyze_features will only use model outputs for identifying important interactions. See Simulation_interaction_discussion.pdf for more details.

Method load_model() in worker.py imports the user-supplied model generation module and returns the model class found in the module. This causes an issue if the user-implemented model.predict() and model.loss() are instance methods (might be because they use and/or need instance variables) rather than static/class methods. Would suggest instantiating model after line 107 e.g. model_inst = model(), and returning the model_inst to cover both scenarios, since model instances can still call static and class methods.

Currently the effect size column in pvalues.csv output file for regression models is empty. We discussed populating it with |MSE(baseline) - MSE(perturbation)|. If a similar field could also be provided for interaction tests, that would be great.