Giter VIP home page Giter VIP logo

abhayjohri23 / datapixie-a-web-scraper Goto Github PK

View Code? Open in Web Editor NEW
0.0 1.0 0.0 67 KB

This repository stores my first ever project using Java. It is a desktop based GUI, backed with Web scrapping algorithm, that scrapes the courses from youtube and udemy (as of now) and sorts them, based on user preference and stores in SQL as well as show case them on a GUI.

Java 100.00%
api api-rest java javaawt javaswing jdbc jdbc-database sql

datapixie-a-web-scraper's Introduction

DataPixie: Unleash Knowledge, Uncover Opportunities.

It brings a desktop based GUI, backed with Web scrapping algorithm, that scrapes the courses from youtube and udemy (as of now) and sorts them, based on user preference and stores in SQL DB and later showcases them on a GUI.

How to use it:

  1. Pre-requisite is that the java files are already present in local repo, and config (pom.xml dependencies etc) are set up.
  2. Set up and load the SQL server via JDBC connector. (Here MySQL 8.0 is used)
  3. You would also need to register to API client 2.0 Affiliate API Program by Udemy and a reliable Youtube API. Registration will get you the public key to access the courses and their details.
  4. Some files are therfore hidden from other developers, you will have to create the requests and add appropriate headers to the requests and use them.
  5. We are good to try the App.

SOP:

  1. First query has to be given in the search bar.
  2. Press the search button (it is a custom Jbutton class, doesn't give look and feel of a button, I know!)
  3. Then Apply the sorting feature. (Optional)
  4. Results will start displaying using the "Next" and "Previous" options in the content panel.

Upcoming features:

  1. Websites other than Youtube and Udemy to be incorporated in next release.
  2. Java Multithreading to be added for processing the requests parallelly and also visit all pages of a website.
  3. Front end GUI to be made more aesthetic and good looking. JavaFX with Scenebuilder to be used.

Adding one snapshot for the viewers for high level understanding of the project! image

Feedbacks will always be apreciated!

datapixie-a-web-scraper's People

Contributors

abhayjohri23 avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.