Proxy changes for same session about crawlee HOT 1 OPEN

Thank you @harm-matthias-harms for bringing this up.

Indeed, there is an issue with the way we're handling the sessions in the browser crawlers. This is because a running browser instance can be reused for multiple requests, but will always have only one proxy URL / session tied to it (because of technical reasons).

We'll try to straighten this up in upcoming patches - in the meantime, you can get the expected behavior by switching the launchContext.useIncognitoPages crawler constructor parameter to true. Note that this tells Crawlee to use a new browser instance for each request, so it can worsen the performance of your crawlers. The actual numbers depend on your use case though.

const crawler = new PlaywrightCrawler({
    launchContext: {
        useIncognitoPages: true, // Use one browser per request, fixes the session pairing issues
    },
    requestHandler: async ({ enqueueLinks, session, proxyInfo }) => {
        ...
    }
});

from crawlee.