[BUG] Firecrawl locally fails to scrape some websites about firecrawl HOT 7 CLOSED

Hey @SalamanderXing, thanks for opening the issue. We're investigating!

from firecrawl.

Hey @SalamanderXing It looks like the playground works well because it uses multiple services and scraping methods with our API. On your setup, you might be using just one method, which is why it’s not pulling up the page.

You can use the Firecrawl API key on your self-hosted setup too. We have a free plan that gives you 500 credits. This should help you get the same results as the playground.

from firecrawl.

@SalamanderXing have you solved the issue? I also have same problem with my self-hosted. Some jobs are basically getting stuck on "Active" queue forever. Also no idea how to delete them.

from firecrawl.

@beydogan I did not. @rafaelsideguide Can you explain more in detail your suggested solution? I was using the api key connected to my local firecrawl. But how come do I need credits for a local setup?

from firecrawl.

@SalamanderXing The playground works well because it uses a mix of services and scraping methods, not just one. This setup helps bypass blockers on websites effectively. For your self-hosted setup, using the Firecrawl API key might help, as it includes access to these multiple services.

You can check out the getScrapingFallbackOrder function in single_url.ts under apps/api/src/scraper/WebScraper for more details on how this works.

from firecrawl.

Rafael, please don't get me wrong, I appreciate the open source and self hosted version of the project, its super useful.

Why would someone need to self-host if they are going to pay for the API anyways?

from firecrawl.

I see, thank you for your reply @rafaelsideguide. Correct me if I'm wrong, the only service among these that is not "hostable" is scrapingbee, right?
Is that the only service that is not available when i run firecrawl locally?

@beydogan That might also answer your question. Someone needs to pay scrapingbee.

from firecrawl.