Parallelisation hangs about axelrod HOT 10 CLOSED

Yup, also hangs here. Only for standard strategies, cheaters and basic go through fine. I am not versed well enough with the code in #122 to tell what the matter is.

from axelrod.

Now I see you are using multiprocessing. My first guess: deadlock caused by a large queue.

from axelrod.

let me tag @meatballs just in case he hasn't seen this...

from axelrod.

I think the problem here, as @langner says, is a deadlocked multiprocessing queue. Specifically, it's the 'done' queue.

I think the problem has been there all along. If you look at lines 92-95 of tournament.py:

       # There is a 0.5 second timeout here as the all_strategies tournament
       # occasionally hangs the join method for some strange reason.
        for process in processes:
            process.join(0.5)

I had to add a timeout to deal with the occasional hang on my machine. That timeout simply causes the process to terminate early and, although it stops the hang, it means the results are mostly an empty matrix. Increasing the the timeout to something like 60s hides the problem completely, but doesn't solve it!

I think the combination of the deterministic cache being passed by #122, the extra strategies added by #121 and the removal of the timeout in #122 have combined to bring the problem to a head.

The crux of the problem is that multiprocessing can't cope with the size of output queue that we are creating. Even setting the queue's maxsize property to 0 (supposedly infinite) doesn't solve the problem.

It might be worth looking at using threading instead of multiprocessing.

from axelrod.

Another potential solution might be to kick off a daemon process first which reads from the done queue and appends to the payoffs list.

from axelrod.

Seems I've fallen for a well known gotcha: https://docs.python.org/2/library/multiprocessing.html#all-platforms

Bear in mind that a process that has put items in a queue will wait before terminating until all the buffered items are fed by the “feeder” thread to the underlying pipe. (The child process can call the cancel_join_thread() method of the queue to avoid this behaviour.)

This means that whenever you use a queue you need to make sure that all items which have been put on the queue will eventually be removed before the process is joined. Otherwise you cannot be sure that processes which have put items on the queue will terminate. Remember also that non-daemonic processes will be joined automatically.

An example which will deadlock is the following:

from multiprocessing import Process, Queue
def f(q):
    q.put('X' * 1000000)
if __name__ == '__main__':
    queue = Queue()
    p = Process(target=f, args=(queue,))
    p.start()
    p.join()                    # this deadlocks
    obj = queue.get()

A fix here would be to swap the last two lines (or simply remove the p.join() line).

from axelrod.

I think I have this fixed. Branch 125 in my repo. I'm still testing it out.

from axelrod.

Works fine for me!

from axelrod.

A relevant SO thread: http://stackoverflow.com/questions/21641887/python-multiprocessing-process-hangs-on-join-for-large-queue (there are also others)

from axelrod.

Fixed by #127

from axelrod.