redsaito / scraping-tutorial Goto Github PK

You want to start scraping? Well this guide will teach you, and not some baby selenium scraping. This guide only uses raw requests and has examples in both python and kotlin. Only basic programming knowlege in one of those languages is required to follow along in the guide.

If you find any aspect of this guide confusing please open an issue about it and I will try to improve things.

If you do not know programming at all then this guide will not help you, learn programming first! Real scraping cannot be done by copy pasting with a vauge understanding.

0. Starting scraping from zero
1. Properly scraping JSON apis often found on sites
2. Evading developer tools detection when scraping
3. Why your requests fail and how to fix them
4. Finding links and scraping videos

Once you've read and understood the concepts behind scraping take a look at a provider in CloudStream. I added tons of comments to make every aspect of writing CloudStream providers clear. Even if you're not planning on contributing to Cloudstream looking at the code may help.

Take a look at Thenos for an example of json based scraping in kotlin.