cui-unige / mcc4mcc Goto Github PK

View Code? Open in Web Editor NEW

4.0 3.0 0.0 6.29 MB

Model Checker Collection for the Model Checking Contest @ Petri nets

Home Page: https://cui-unige.github.io/mcc4mcc/

License: MIT License

Python 51.02% Makefile 9.87% Shell 29.94% Dockerfile 9.18%

model-checking petri-net pnml

mcc4mcc's People

Contributors

Stargazers

Watchers

mcc4mcc's Issues

Experiment with data

Output the decision trees

It may be interesting to look at the shape of the tree in order to identify useful or useless characteristics.

Maybe the "don't care" pattern of decision diagrams appears!

Refactor code into a real python module

Add new scoring functions

For instance:

linearly decreasing of time
linear decreasing of memory
non-linear functions.

Add Spot

Add continuous integration

pycodingstyle on python code;
autopep8 if needed;
pylint on python code;
tests.

Problem with `custom-algo`

When i run:

$ ./extract.py --iterations=1 --duplicates=false

I get the following error. It does not happen with other algorithms.

INFO: Learning using algorithm: 'custom-algo'.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  4.13it/s]
INFO: Algorithm: custom-algo
INFO:   Min     : 0.49337490257209665
INFO:   Max     : 0.49337490257209665
INFO:   Mean    : 0.49337490257209665
INFO:   Median  : 0.49337490257209665
Traceback (most recent call last):
  File "./extract.py", line 685, in <module>
    analyze_learned()
  File "./extract.py", line 667, in analyze_learned
    SCORES[name] = mcc_score(algorithm)
  File "./extract.py", line 566, in mcc_score
    alg_or_tool.predict(pandas.DataFrame([test]))[0]
  File "./extract.py", line 123, in predict
    y_pred[mask] = self.multi.predict(test_x[mask])
  File "/usr/local/lib/python3.6/site-packages/sklearn/ensemble/forest.py", line 538, in predict
    proba = self.predict_proba(X)
  File "/usr/local/lib/python3.6/site-packages/sklearn/ensemble/forest.py", line 578, in predict_proba
    X = self._validate_X_predict(X)
  File "/usr/local/lib/python3.6/site-packages/sklearn/ensemble/forest.py", line 357, in _validate_X_predict
    return self.estimators_[0]._validate_X_predict(X, check_input=True)
  File "/usr/local/lib/python3.6/site-packages/sklearn/tree/tree.py", line 373, in _validate_X_predict
    X = check_array(X, dtype=DTYPE, accept_sparse="csr")
  File "/usr/local/lib/python3.6/site-packages/sklearn/utils/validation.py", line 462, in check_array
    context))
ValueError: Found array with 0 sample(s) (shape=(0, 23)) while a minimum of 1 is required.

Review bad performance of learning algorithms

For @mencattini , in branch issue-7.

Add regression algorithms

Fix removed code that should not

@mencattini look at 17f9b47, it seems that you removed some of my code, that was about distance between tools. Is it intentional?

Rework data. Need much more data to improve the accuracy.

Select tools that are within a % of the best, instead of the best only.

Mount vmdk images to build docker containers

See for instance this post.

Add scoring function from the MCC

It should compute the score given the scoring algorithm of the model checking contest.

Add StrataGEM

Import 2018 results

Switch the code to Pep8 standard.

It's more a request than an issue. Can we convert the code to the Pep8 standard. It's a syntaxe standard used in most of the python project.
The errors to correct are :

E272 -> multiple spaces before keyword
E201 -> whitespace after '['
E202 -> whitespace before ']'
E251 -> unexpected spaces around keyword/parameters
E211 -> whitespaces before '('

Update all docker images

And also improve the mcc-head scripts.

Fix the distance definition

diff = x_1 - x_2
return max(diff, 2)

Add voting between classifier results

Try to enhance results by voting among the results of several algorithms.

Negative score values for regression

When i run:

$ ./extract.py

The results for regression algorithms are:

INFO: Algorithm: decision-tree-regression
INFO:   Min     : -0.23408958008379654
INFO:   Max     : -0.013248075792894998
INFO:   Mean    : -0.10488969127417243
INFO:   Median  : -0.10611363409432173
INFO:   Stdev   : 0.06583227749288169
INFO:   Variance: 0.004333888759899777
INFO: Learning using algorithm: 'knn-regression'.

INFO: Algorithm: knn-regression
INFO:   Min     : -0.10861812564105898
INFO:   Max     : 0.030489299735053432
INFO:   Mean    : -0.04368667835772075
INFO:   Median  : -0.05275172799947181
INFO:   Stdev   : 0.04813321043108172
INFO:   Variance: 0.002316805946402794
INFO: Learning using algorithm: 'random-forest-regression'.

INFO: Algorithm: random-forest-regression
INFO:   Min     : -0.05989802856982962
INFO:   Max     : 0.06089070070317171
INFO:   Mean    : 0.003811517055560587
INFO:   Median  : -0.003230270254699741
INFO:   Stdev   : 0.03777773047303064
INFO:   Variance: 0.0014271569196929476

The problem only happens with the option duplicates=false.

Compute mean distance between the best algorithm and the best results

Less data used

@saucisson
I have noticed there are less datas than before.
INFO: Select 9366 best entries.
And before it was :
INFO: Select 10261 best entries.

Is it normal ?

Compare `mcc4mcc` performance with virtual machines

Store distances together with tools

The vector for learning algorithms could contain a distance from the fastest tool, together with the tool name. Can it allow us to find the fastest tool instead of just a correct tool?

Add a script to test integration of tools

Identify which characteristics are interesting

We could try to find the interesting characteristics, by removing each characteristic and evaluating the algorithms.

cui-unige / mcc4mcc Goto Github PK

mcc4mcc's People

Contributors

Stargazers

Watchers

mcc4mcc's Issues

Recommend Projects

Recommend Topics

Recommend Org