Cannot start shared-memory branch on Windows about botogram HOT 19 CLOSED

Toying around, I replaced the SharedMemoryManager 's _manager attribute with multiprocessing.Manager() and arrived at a different, possibly helpful error:

20:03.53 -   DEBUG   - Component botogram just loaded into the bot
20:03.55 -   DEBUG   - Component botogram just loaded into the bot
20:03.57 -   INFO    - The botogram runner is booting up.
20:03.57 -   INFO    - Press Ctrl+C to exit.
Traceback (most recent call last):
...
\runner\__init__.py", line 56, in run
    self._shared_memory.start()
  File "C:\.....\botogram\runner\shared.py", line 60, in start
    self._manager.start()
  File "C:\Python34\Lib\multiprocessing\managers.py", line 475, in start
    assert self._state.value == State.INITIAL
AssertionError

Brad

from botogram.

Just a random note, I've figured out how to simulate the Windows behavior on my Linux machine, so it should be easier for me to simulate what's going wrong.

from botogram.

Toying around, I replaced the SharedMemoryManager 's _manager attribute with
multiprocessing.Manager() and arrived at a different, possibly helpful error:

No, that's because Manager() automatically calls the start() method, and obviously you can't start an already-started manager.

from botogram.

Ah, I see that now.

I've figured out how to simulate the Windows behavior on my Linux machine,

Very good, no doubt this will be more efficient than my actions. :)

from botogram.

I think I've found the problem (hopefully the only one).

When the runner's shared memory driver is serialized, it requires to the master process a new communication queue. Unfortunately, the master process starts accepting shared memory's commands after all processes are booted up. This doesn't cause any problem on Linux and OSX, but on Windows the driver object is serialized while booting up processes, so a deadlock happens.

Now I'm just tired, tomorrow I'll see how I can fix this problem (I wish this doesn't require a brand new process just for this thing).

Pietro.

from botogram.

Just a quick note, I've managed to get over the the deadlock. Unfortunately, a new error came out just now :-(

from botogram.

That's good news. I'm interested in seeing how you got around it.

from botogram.

I've pushed 10833fa, which doesn't bring full compatibility (there is the error I mentioned yesterday), but resolves the ones you reported I was able to reproduce. Can you try directly on Windows which errors it's giving?

from botogram.

Here it is:

05:08.19 -   DEBUG   - Component botogram just loaded into the bot
05:08.21 -   INFO    - The botogram runner is booting up.
05:08.21 -   INFO    - Press Ctrl+C to exit.
05:08.22 -   DEBUG   - Component botogram just loaded into the bot
05:08.24 -   DEBUG   - Component botogram just loaded into the bot
05:08.26 -   DEBUG   - Shared process is ready! (pid: 7048)
Traceback (most recent call last):
  File "C:\Python34\Lib\multiprocessing\queues.py", line 242, in _feed
    obj = ForkingPickler.dumps(obj)
  File "C:\Python34\Lib\multiprocessing\reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
  File "C:\Python34\Lib\multiprocessing\queues.py", line 57, in __getstate__
    context.assert_spawning(self)
  File "C:\Python34\Lib\multiprocessing\context.py", line 347, in assert_spawning
    ' through inheritance' % type(obj).__name__
RuntimeError: Queue objects should only be shared between processes through inheritance

from botogram.

Perfect. Later in the afternoon I'll try to migrate from queues to pipes.

from botogram.

Current status: I can't find a way to transfer pipes to different processes via other pipes. Uff.

from botogram.

Hmm, seems to be a tough problem. I have set up the docker tool chain in order to further develop. Perhaps with time a solution to this process issue will present itself. Thank you for looking into it nonetheless.

Brad

from botogram.

Yesterday I started implementing the interprocess communication with standard sockets. If this doesn't work (but I think it will) I'll probably drop Windows suppory, since I'm wasting too much time on this, delaying new features.

from botogram.

Completely understandable. I'm looking forward to what else you come up with. Cheers.

from botogram.

Seems that the interprocess communication with sockets works fine. Now I'll start figuring out how to correctly use it with the multiprocessing's Managers, but I'm optimistic this time.

from botogram.

By the way, I'm a bit busy, but I'm still working on this :)

from botogram.

Hey @tsculpt, do you mind checking if the latest commit on the socket-ipc branch is working?

Thanks,
Pietro.

from botogram.

A friend of mine tested the changes on Windows, and all works flawlessy!

from botogram.

Awesome work Pietro! I'm looking forward to release. Cheers.

from botogram.