Create drafttek prospect scraper about colly-draft-prospects HOT 6 CLOSED

Copied scraper here. It's still in need of significant changes, as the other page was wildly different looking, even if the content was the same.

from colly-draft-prospects.

Some changes have been made, but the scraper is still in a debug state. There are two tables on the web page, and we only need data from the second one, but we're getting data from both tables. I'm sure it has something to do with how c.OnHTML("tr td:nth-of-type(4)" and the similar collectors are working, but I don't have a ton of experience with the DOM, so I'm going to need to do some more troubleshooting.

from colly-draft-prospects.

#2 is a great start. It looks like OnHTML(...) uses GoQuery, so I'll need to read up on that.

Link to GoQuery:
https://github.com/PuerkitoBio/goquery

from colly-draft-prospects.

#3 adds the html element without goquery to the scraper, since it appears that goquery may or may not be necessary to complete this task.

This article shows an implementation that uses colly both with goquery and with html.
https://benjamincongdon.me/blog/2018/03/01/Scraping-the-Web-in-Golang-with-Colly-and-Goquery/

from colly-draft-prospects.

#4 tries to use the data obtained in statements like c.OnHTML("tr td:nth-of-type(4)" and puts them into slices. Rather than trying to filter out the data as it goes into the slice, we can make the assumption that the only garbage data is coming from the start of the slice, so if we look for the category header, we may be able to just write all the data that follows to the csv.

from colly-draft-prospects.

This is fixed by #5.

from colly-draft-prospects.