No links are being enqueued on some pages about crawlee HOT 3 CLOSED

Hello, @batu-archive! (and sorry for the wait).

Indeed, this page is a bit weird - navigation is fully JS-based, so enqueueLinks cannot do much here. Unfortunately, it also uses some more-complicated-than-usual logic for tab management, so the enqueueLinksByClickingElements doesn't work out of the box either (although this method should work with opening popup tabs too).

I found one thing when playing with DevTools on the page - it loads all the (important) data with XHR requests from https://www.metacareers.com/graphql. With the preNavigationHooks and page.route() you can do something like this:

const crawler = new PlaywrightCrawler({
  requestHandler: router,
  preNavigationHooks: [async ({ page, sendRequest, enqueueLinks, request: pageRequest }) => {
        // do this only on the first page
        if(pageRequest.url !== 'https://www.metacareers.com/jobs') return; 

        // catch the XHR requests to the API endpoint
        await page.route('https://www.metacareers.com/graphql', async route => {
            const request = await route.request();

            // check if we caught the correct API request (with all the job data)
            if(request.headers()['x-fb-friendly-name'] === 'CareersJobSearchResultsQuery') { 
                
                // if so, we make the request again separately
                const data = await sendRequest({ 
                    url: request.url(),
                    method: request.method() as any,
                    headers: request.headers(),
                    body: request.postData() as any,
                }).then(x => x.body)

                const openings = JSON.parse(data).data.job_search; // get the data

                // enqueue all of the urls of the job openings
                await enqueueLinks({
                    urls: openings.map(x => new URL(x.id, 'https://www.metacareers.com/jobs/').toString()), 
                });
            }

            await route.continue(); // resume the original API request
        })
  }],
  headless: false,
});

The cool thing is - you don't need pagination here, as all of the job data (all 1600+ positions) are loaded in the first and only API request.

Another thing I also noticed is that the separate job pages also load without JS - so you might get much better performance if you process these requests with the CheerioCrawler (this I didn't test though, it's possible that the server might block a request from a non-browser HTTP client).

We won't make any adjustments to the enqueueLinks methods just now, as this is a really specific case... but I'd love to hear more about your use case!

Did the code above help? I'll close this issue as wontfix now, but in case you have any additional questions - fire away!

Thanks!

from crawlee.

I have found a way looking at source code which is enqueueLinksByClickingElements imho. However I think this method listens for navigation in current page context. In my case when we click on job ad, it tries to open new tab which is immediately closed in ms for some reason

from crawlee.

Hi @barjin thank you very much for detailed answer ❤️💎🫶 and yes it definitely helped even though I was looking for "how to fish" rather than "getting fish ready" 😅

from crawlee.