Giter VIP home page Giter VIP logo

scraping-tutorial's Introduction

Requests based scraping tutorial

You want to start scraping? Well this guide will teach you, and not some baby selenium scraping. This guide only uses raw requests and has examples in both python and kotlin. Only basic programming knowlege in one of those languages is required to follow along in the guide.

If you find any aspect of this guide confusing please open an issue about it and I will try to improve things.

If you do not know programming at all then this guide will not help you, learn programming first! Real scraping cannot be done by copy pasting with a vauge understanding.

  1. Starting scraping from zero
  2. Properly scraping JSON apis often found on sites
  3. Evading developer tools detection when scraping
  4. Why your requests fail and how to fix them
  5. Finding links and scraping videos

Once you've read and understood the concepts behind scraping take a look at a provider in CloudStream. I added tons of comments to make every aspect of writing CloudStream providers clear. Even if you're not planning on contributing to Cloudstream looking at the code may help.

Take a look at Thenos for an example of json based scraping in kotlin.

scraping-tutorial's People

Contributors

arjixwastaken avatar blatzar avatar justchokingaround avatar justfoolingaround avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

scraping-tutorial's Issues

needing help with scraping providers

I learnt much from your scraping tutorial but I'm having issue in retrieving the actual stream links from the provider such as from Megacloud I need to decrypt the string using keys and iv which I don't know where to search for and how.

Thanks for your repo !

librewolf ?

librewolf.console.logging_disabled

I searched but it doesn't show up, where is it?

version 120.0.1-1

Need Help with Scraping

Hey, have read your well-structured documentation on web scraping.

I am a JavaScript developer who uses axios and cheerio to web scrape (They work just like (jsoup in Kotlin)

I have a hard time extracting video in movies sites like sflix.to or zoro.to and I was asking if you make try to make me understand how it's done.

I have seen the source code of CloudStream but I don't know Kotlin so I have a hard time understanding what the code does and how to follow the execution of the program.

I have tried researching on this but to no avail so your assistance will high be appreciated.

Regards.

(NOT AN ISSUE) Just want to Thank U

Hey @Blatzar , Thanks for making an awesome repository, i found ur repo from ani-cli repo, this repo is so underated, i wish it had 5K+โญ so that it comes on top of the search results, when searched for scraping. all other repos have not explained in such detailed manner. Thank u once again for making such a wonderful repo. ๐Ÿฅ‚

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.