Archipelagos and the mp_island namespace about pagmo2 HOT 11 CLOSED

This is due to the behaviour of the serialization library we are using (dill). There's a few possible fixes:

1. do the imports within the methods (as you mentioned already)

2. do the imports as part of the class definition, and then employ this slightly kludgy way of using them:

class toy_problem:
     import numpy as _np

     def fitness(self, x):
         return [type(self)._np.sum(type(self)._np.sin((x-.2)**2))]

     def get_bounds(self):
         return (type(self)._np.array([-1] * 3), type(self)._np.array([1] * 3))

3. Set the "recurse" option for dill:

import dill
dill.settings['recurse'] = True

import numpy as np

class toy_problem:
    ...

from pagmo2.

Oh thanks, I had been thinking it was an issue with multiprocessing itself. Is this a significant enough issue that I could try to work on a quick documentation update, perhaps under Python Tutorials/Use of the class archipelago?

from pagmo2.

It would be good to have a section in the documentation illustrating these types of pitfalls, so feel free to go ahead if you like :)

For context, we decided to go with an external serialization library (dill) because, contrary to the standard pickle module, it allows also to serialize classes/functions/etc. defined interactively (note that pagmo also supports standard pickling, it just uses dill instead of pickle when it needs to distribute data to external processes). A library similar to dill is cloudpickle, and we could consider eventually adding support for more serialization protocols if the need arises. For the moment, though, we need more experience before taking a decision like this.

from pagmo2.

I'm back with a similar issue already!

import pygmo as pg
import numpy as np
import dill

dill.settings['recurse'] = True

class toy_problem:

    def fitness(self, x):
     return [np.sin(x[0] + x[1] - x[2])]

    def gradient(self, x):
     return pg.estimate_gradient(lambda x: self.fitness(x), x)

    def get_bounds(self):
     return (np.array([-1] * 3), np.array([1] * 3))


if __name__=='__main__':
    mma = pg.algorithm(pg.nlopt("mma"))
    p_toy = pg.problem(toy_problem())
    archi = pg.archipelago(n=6, algo=mma, prob=p_toy, pop_size=1)
    archi.evolve()
    archi.wait_check()

dies with

KeyError: '__builtins__

in worker->_evolve_func->estimate_gradient
It's a nlopt problem with estimate_gradient, and the gradient-less nlopt algorithms run fine. Presumably also a serialization problem?
Thanks for the help, I promise to update the docs when I get everything fully working :)

from pagmo2.

Issue confirmed on my system.

from pagmo2.

here is the full stack

RuntimeError                              Traceback (most recent call last)
<ipython-input-1-38e735f44774> in <module>()
     22     archi = pg.archipelago(n=6, algo=mma, prob=p_toy, pop_size=1)
     23     archi.evolve()
---> 24     archi.wait_check()
     25 

RuntimeError: The asynchronous evolution of a Pythonic island of type 'Multiprocessing island' raised an error:
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/dario/.local/lib/python3.6/site-packages/pygmo/_py_islands.py", line 40, in _evolve_func
    return algo.evolve(pop)
  File "<ipython-input-1-38e735f44774>", line 13, in gradient
KeyError: '__builtins__'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/dario/.local/lib/python3.6/site-packages/pygmo/_py_islands.py", line 128, in run_evolve
    return res.get()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 608, in get
    raise self._value
KeyError: '__builtins__'

from pagmo2.

The error is somehow linked to the use of:

    def gradient(self, x):
     return pg.estimate_gradient(lambda x: self.fitness(x), x)

as it disappears trying for example pygmo.de() instead of the nlopt

from pagmo2.

It is connected to the use of the lambda function in the estimate_gradient call. For some reason, the recurse option of dill causes this issue. If you don't use recurse, and do the imports within the methods, it works as expected. Testing things out right now to see if there are better workarounds.

from pagmo2.

@mishpat We are testing out a cloudpickle as an alternative to dill in #108. My local tests indicate that it handles both the examples in this report without any extra setup. If it works fine in the CI, we'll merge the PR and do a new pagmo release shortly afterwards.

from pagmo2.

@mishpat we have released version 2.3 which includes the change to cloudpickle. Packages should be up in a few hours. Let us know if it works for you.

from pagmo2.

Everything seems to work great now, thanks!

from pagmo2.