Giter VIP home page Giter VIP logo

Comments (3)

erdewit avatar erdewit commented on June 2, 2024

Not sure where the strange behavior stems from. But generally one can expect the strangest things with ProcessPool with the Unix default of process forking. It will fork the global state into the child processes. When using the spawn method for child processes, the problem becomes serialization of functions as used by the task.

I would suggest to use something else, for example the vastly superior (ahem) distex Pool. This simplifies the code and it becomes possible to run tasks on any other machine that happens to run a SSH server:

import asyncio
from typing import List

import distex

import nest_asyncio
nest_asyncio.apply()


async def get_urls_httpx(urls):
    import httpx
    import asyncio
    
    async def _get_url_httpx(url, session):
        resp = await session.get(url, timeout=None)
        return resp.status_code

    session = httpx.AsyncClient()
    try:
        coros = [_get_url_httpx(url, session) for url in urls]
        result = await asyncio.gather(*coros)
    finally:
        await session.aclose()
    return result


async def get_urls_aiohttp(urls: List[str]):
    import asyncio
    import aiohttp
    
    async def _get_url_aiohttp(url, session):
        async with session.get(url) as resp:
            return resp.status
    
    async with aiohttp.ClientSession() as session:
        coros = [_get_url_aiohttp(url, session) for url in urls]
        result = await asyncio.gather(*coros)
    return result
        

def get_all_urls(all_urls: List[List[str]], func):
    with distex.Pool() as pool:
        results = list(pool.map(func, all_urls, ordered=False))
    return results

from nest_asyncio.

dotsdl avatar dotsdl commented on June 2, 2024

Thanks for this @erdewit! I'll consider alternative approaches, in particular non-forking approaches for the process pool.

from nest_asyncio.

dotsdl avatar dotsdl commented on June 2, 2024

For the record: switching get_all_urls above to force use of 'spawn' as multiprocessing context doesn't show the problem this issue raises:

def get_all_urls(all_urls: List[List[str]], func):
    import multiprocessing as mp
    from concurrent.futures import ProcessPoolExecutor, as_completed

    ctx = mp.get_context('spawn')

    with ProcessPoolExecutor(mp_context=ctx) as executor:
        futures = [executor.submit(func, urls) for urls in all_urls]

        print("waiting on futures")
        results = []
        for future in as_completed(futures):
            print("future", future)
            result = future.result()
            results.append(result)

    return results

from nest_asyncio.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.