Giter VIP home page Giter VIP logo

colly-draft-prospects's Introduction

colly-draft-prospects

Source code for web scraping football draft prospects from the DraftTek NFL Big Board.

The scraper is written in Golang and uses the Colly scraper. The binary file in the repo is compiled for Linux, but it could be compiled to use in a different operating system if needed.

Once the ranks have been scraped, I use csvkit to merge all of the files and join them together with information about the locations of the schools. The csvkit commands are in the csvkitcommands.txt files.

Once the ranks have been assembled, I use OpenRefine to clean the data for consistency. The data cleaning steps are contained in openRefineDataMerge.json.

colly-draft-prospects's People

Contributors

justintm avatar ulziibuyan avatar zo0o0ot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

colly-draft-prospects's Issues

Modify scraper to use date on webpage in CSV instead of current date

On page 1, there's a date signifying the last time the rankings were updated.

The inner HTML looks something like this:
(October 14, 2018) 2019 NFL Draft Big Board: Top-300
CSS Selector: #calloutwifnba > strong:nth-child(1)
CSS Path: html body div#outer div#wrapper div#content div#calloutwifnba strong
XPath: /html/body/div[1]/div[3]/div[2]/div/strong

Get the date from the page. Put it in the format YYYY-MM-DD and write it in the output CSV instead of the current date. Keep the Current Date output for the CSV file name.

Create drafttek prospect scraper

Web page is https://www.drafttek.com/Top-100-NFL-Draft-Prospects-2019.asp

There are up to four pages total. Currently, three of the four have data.
https://www.drafttek.com/Top-100-NFL-Draft-Prospects-2019.asp
https://www.drafttek.com/Top-100-NFL-Draft-Prospects-2019-Page-2.asp
https://www.drafttek.com/Top-100-NFL-Draft-Prospects-2019-Page-3.asp

Eventually, this page will likely have content:
https://www.drafttek.com/Top-100-NFL-Draft-Prospects-2019-Page-4.asp

Desired output is similar to https://github.com/Leagify/colly-draft-prospects/blob/master/scraper.go but the page formatting has changed so some of the code will be different.

Output desired (probably as a CSV):
Rank, Name, School, P1, Date

Refine data programatically.

Using something like OpenRefine, refine the scraped data for consistency to minimize any cleaning and grouping that needs to be done so that this data needs to be charted.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.