Comments (3)
Thanks for you help, the problem was that I was not doin p.clear()
- for some reason did not see it anywhere in examples
I am using it in python 2.7 and instead of ValueError: Pool not running
in your example was getting AssertionError
from pathos.
I don't see an AssertionError
from similar test code. Can you please provide some self-contained small example that produces the behavior you are seeing? Also provide your version of python and of pathos.
from pathos.
Note that a Pool
in pathos
is a singleton, so if you close a Pool
and try to use it again it is designed to fail unless you also clear
if from memory. Each unique pool has an ID that is registered in a global hashmap.
Here's the expected behavior:
Python 3.8.18 (default, Aug 25 2023, 04:23:37)
[Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pathos.pools as pp
>>> p = pp.ProcessPool(4) # create a pool
>>> p.map(lambda x:x*x, range(4))
[0, 1, 4, 9]
>>> p.close()
>>> p.join()
>>>
>>> p = pp.ProcessPool(4) # grabs the existing closed pool
>>> p.map(lambda x:x*x, range(4))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/mmckerns/lib/python3.8/site-packages/pathos/multiprocessing.py", line 154, in map
return _pool.map(star(f), zip(*args), **kwds)
File "/Users/mmckerns/lib/python3.8/site-packages/multiprocess/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/Users/mmckerns/lib/python3.8/site-packages/multiprocess/pool.py", line 473, in _map_async
self._check_running()
File "/Users/mmckerns/lib/python3.8/site-packages/multiprocess/pool.py", line 350, in _check_running
raise ValueError("Pool not running")
ValueError: Pool not running
>>> p.clear() # clear the pool from memory
>>> p.map(lambda x:x*x, range(4))
[0, 1, 4, 9]
>>> p.close()
>>> p.join()
>>> p.clear() # clear the pool from memory
>>>
>>> p = pp.ProcessPool(4)
>>> p.map(lambda x:x*x, range(4))
[0, 1, 4, 9]
>>> p.close()
>>> p.join() # not cleared from memory
>>>
>>> p = pp.ProcessPool(2) # create a different new pool
>>> p.map(lambda x:x*x, range(4))
[0, 1, 4, 9]
>>> p.close()
>>> p.join()
>>> p.clear()
>>>
from pathos.
Related Issues (20)
- Support of PyTorch Tensors on CPU HOT 4
- Getting "TypeError: can't pickle _cffi_backend.FFI objects" on "results.get()" HOT 5
- Code corruption with Pool? HOT 6
- add CI support for pypy3.8, pypy3.9, and python3.12
- add formal support for python3.11
- TypeError: no default __reduce__ due to non-trivial __cinit__ HOT 2
- A closed pool may be returned while calling `ProcessPool.__init__` HOT 2
- How to run multiple processes in parallel using Pathos? HOT 5
- pathos Pool can't find dynamically generated log function HOT 2
- add changelog to release notes HOT 1
- Pathos still giving pickle error HOT 3
- Shared Memory Objects HOT 1
- host release-specific docs so docs match what's available HOT 3
- [multiprocessing] Propagate OOM errors
- pathos.pools.ProcessPool deadlock/hang on exceptions HOT 5
- map raises DeprecationWarning in 3.12.0b1 HOT 1
- Endless warning message when use imap
- Information about load balancing/resource allocation HOT 2
- it is possible to use pathos with gmpy2 ? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pathos.