fi-mihej / interprocesspyobjects Goto Github PK

Hey, I am facing an issue using Cengal/this library (SharedMemory seems to mainly come from Cengal) and I want to reach out via Discord (or email), but the link in the README does not work.

Hi,

I am attempting to use this library according to the pattern provided in the very helpful sender.py and receiver.py examples, but am running into an issue.

My application is async and so I want to use the async semantics provided by the library. That said, while I am reasonably experienced with Python, this is my first foray into async, so it's possible that my problem below results from something I am doing wrong.

The issue is that when I try to run my code according to the example in receiver.py, as follows:

        async with ashared_memory_context_manager.if_has_messages() as shared_memory: 

I receive an error indicating:

AttributeError: 'ASharedMemoryManager' object has no attribute 'if_has_messages'. Did you mean: 'has_messages'?

If I try using the has_messages method, like this:

async with self._ashared_memory_manager.has_messages() as shared_memory:

I get this error:

TypeError: 'coroutine' object does not support the asynchronous context manager protocol

I am grateful for any help you can provide.

Many thanks -- I appreciate your work on this project!
--Phil

Hi Mihej,

Thank you for your contributions to creating this repo, I'm trying to make use of it in the following scenario, which I think may help others as well so I'm creating an issue here:

I'm writing a PyTorch program that involves training on a large dataset, and I want to cache the whole dataset to memory to speed up training. However, since the dataset is large and I want to run multiple hyperparameter settings at the same time and the same dataset, I want to share the cached dataset across different runs. Could you give a simple example of how I can achieve this?

I think a SharedMemory class and the put_message(dataset), take_message(dataset) methods should work, but I'm not sure if I can call the take_message multiple times across different runs once I've put it.

The dataset I want to cache has the following form:

cached_dataset = [
data_1, # dictionary of several numpy arrays like images, etc
data_2,
...,
]

Do you have any advice on how I can improve the reading performance of the shared dataset? And after I've put the cached dataset into the shared memory, is it safe for me to delete the original cached dataset so that it won't take up twice of the memory?

Thank you for the help!