mbelmadani / motifgp Goto Github PK

Modify engine.py to evaluate of solutions in parallel. (add toolbox.register("map", some_mapping_function)).

Possibilities include python multiprocessing, Scoop, or pathos. The multiprocessing module is standard python, while Scoop appears to integrate well with DEAP and seems to extend beyong multiprocessing (e.g support for clusters), which is nice. The issue with these modules is that they don't automatically serialize instance methods and certain modules, required by the function mapped to the toolbox.evaluate method, which makes it difficult to use with motifgp without significant modifications. Pathos appears to handle serialization by default, so it might a good start point.

An issue with this multiprocessing is that it might break memoization since the dict for the cache is not shared, so it probably won't be updated or shared between evaluations. This might be possible to fix with a workaround the shared memory using. If not, it might be better to add cache-less parallelism, or reconsider python multiprocessing or scoop.

Duplicate

Standard output (messages) writing is currently used inconsistently (for example, sometimes using sys.write(), others with print() ).

1. Every message or result specifc output (like the output pareto front, or motif statistics) should be using print() function. Other messages should use the python 'logging' module. Logger.info() for messages that are expected, Logger.critical() when the program errors and terminates, and Logger.warning( ) for anything else.

2. At the same time, add a check for a verbosity level for each Logger and print. Having the verbosity as a string would allow more flexibility to the user. An idea:

i = info
w = warning
s = stats
r = results
d = debug - Also print the source location of the message

The user provides a string with these tags (e.g. "isr" for everything except warnings) and logging is only done for those in the string. Critical level messages are always printed, unless there's a good reason not to.

Logger should be configured as:

import logging

if "l" in options.verbosity:
    FORMAT = '[%(filename)s:%(lineno)s - %(funcName)20s() ] %(asctime)15s - %(levelname)s : %(message)s'
else:
    FORMAT = '%(asctime)15s - %(levelname)s : %(message)s'
logging.basicConfig(format=FORMAT)
logger = logging.getLogger(__name__)

and messages logged with:

if` "w" in options.verbosity:
    logger.warning("The warning message")

This change should not be made for standalone scripts, this is only for motifgp.py and its classes.