leagify / colly-draft-prospects Goto Github PK

I just heard about the tool called q.

The homepage is https://harelba.github.io/q/

I have csvkit commands in the data handling, but q might be a better option.

Consider doing this.

An OpenRefine API exists:

https://github.com/OpenRefine/OpenRefine/wiki/OpenRefine-API

I've figured out a workaround by setting up something with VisualPing that emails me when the page changes, but it would be nice if the process was automated.

I should probably learn how to do tagged releases when I do this.

Using something like OpenRefine, refine the scraped data for consistency to minimize any cleaning and grouping that needs to be done so that this data needs to be charted.

On page 1, there's a date signifying the last time the rankings were updated.

The inner HTML looks something like this:
(October 14, 2018) 2019 NFL Draft Big Board: Top-300
CSS Selector: #calloutwifnba > strong:nth-child(1)
CSS Path: html body div#outer div#wrapper div#content div#calloutwifnba strong
XPath: /html/body/div[1]/div[3]/div[2]/div/strong

Get the date from the page. Put it in the format YYYY-MM-DD and write it in the output CSV instead of the current date. Keep the Current Date output for the CSV file name.

Web page is https://www.drafttek.com/Top-100-NFL-Draft-Prospects-2019.asp

There are up to four pages total. Currently, three of the four have data.
https://www.drafttek.com/Top-100-NFL-Draft-Prospects-2019.asp
https://www.drafttek.com/Top-100-NFL-Draft-Prospects-2019-Page-2.asp
https://www.drafttek.com/Top-100-NFL-Draft-Prospects-2019-Page-3.asp

Eventually, this page will likely have content:
https://www.drafttek.com/Top-100-NFL-Draft-Prospects-2019-Page-4.asp

Desired output is similar to https://github.com/Leagify/colly-draft-prospects/blob/master/scraper.go but the page formatting has changed so some of the code will be different.

Output desired (probably as a CSV):
Rank, Name, School, P1, Date