Comments (3)
Not sure where the strange behavior stems from. But generally one can expect the strangest things with ProcessPool with the Unix default of process forking. It will fork the global state into the child processes. When using the spawn method for child processes, the problem becomes serialization of functions as used by the task.
I would suggest to use something else, for example the vastly superior (ahem) distex Pool. This simplifies the code and it becomes possible to run tasks on any other machine that happens to run a SSH server:
import asyncio
from typing import List
import distex
import nest_asyncio
nest_asyncio.apply()
async def get_urls_httpx(urls):
import httpx
import asyncio
async def _get_url_httpx(url, session):
resp = await session.get(url, timeout=None)
return resp.status_code
session = httpx.AsyncClient()
try:
coros = [_get_url_httpx(url, session) for url in urls]
result = await asyncio.gather(*coros)
finally:
await session.aclose()
return result
async def get_urls_aiohttp(urls: List[str]):
import asyncio
import aiohttp
async def _get_url_aiohttp(url, session):
async with session.get(url) as resp:
return resp.status
async with aiohttp.ClientSession() as session:
coros = [_get_url_aiohttp(url, session) for url in urls]
result = await asyncio.gather(*coros)
return result
def get_all_urls(all_urls: List[List[str]], func):
with distex.Pool() as pool:
results = list(pool.map(func, all_urls, ordered=False))
return results
from nest_asyncio.
Thanks for this @erdewit! I'll consider alternative approaches, in particular non-forking approaches for the process pool.
from nest_asyncio.
For the record: switching get_all_urls
above to force use of 'spawn' as multiprocessing context doesn't show the problem this issue raises:
def get_all_urls(all_urls: List[List[str]], func):
import multiprocessing as mp
from concurrent.futures import ProcessPoolExecutor, as_completed
ctx = mp.get_context('spawn')
with ProcessPoolExecutor(mp_context=ctx) as executor:
futures = [executor.submit(func, urls) for urls in all_urls]
print("waiting on futures")
results = []
for future in as_completed(futures):
print("future", future)
result = future.result()
results.append(result)
return results
from nest_asyncio.
Related Issues (20)
- 1.5.2 and 1.5.3 broke some current_task detection HOT 2
- 1.5.1-1.5.4 is breaking vaex due to #61 HOT 3
- nested tasks causing deadlock? HOT 1
- Request stuck in ProcessPoolExecutor HOT 4
- feature request: unapply HOT 1
- DeprecationWarning on Python 3.10
- RuntimeError: cannot enter context: <_contextvars.Context object at %> is already entered HOT 1
- ThreadPoolExecutor issue HOT 8
- Release tag for v1.5.6 HOT 1
- Execution of Unit Tests Dead Locked After Upgrade to Python 3.10.7 HOT 3
- Lost Logs HOT 1
- nest_ayncio breaks tests under Python 3.10.9 HOT 2
- nest_asyncio.apply not working unless entire asyncio package is imported HOT 1
- RFE: is it possible to start making github releases?🤔 HOT 2
- Nested asyncio.run is parallel on first call but serial on second HOT 1
- How to share websocket context in quart?
- After `apply()` and `asyncio.run()`, Jupyter kernel cannot execute any code HOT 7
- Collision with AsyncIO HOT 1
- `nest_asyncio` breaks `discord.py`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nest_asyncio.