albertsuarez / azlyrics-scraper Goto Github PK

The row in azlyrics_lyrics_l.csv looks like:

"liz phair","https://www.azlyrics.com/p/phair.html","6'1"","https://www.azlyrics.com/lyrics/lizphair/61.html","i bet you fall in bed[....]"

There's an extra double-quote in the song title field, which confuses the parser in Python's csv library (and probably most others). Per the csv RFC:

If double-quotes are used to enclose fields, then a double-quote
appearing inside a field must be escaped by preceding it with
another double quote. For example:
"aaa","b""bb","ccc"

(btw, thank you for publishing this dataset! It's sorely needed.)

In the latest release of the dataset, there are 74 rows corresponding to Liz Phair songs. 61 of those rows are in azlyrics_lyrics_l.csv under the artist name "Liz Phair". 13 are in azlyrics_lyrics_p.csv under "Phair, Liz".

There are 11 songs which appear in both files. As far as I can tell, the lyrics, song url, and song title are identical between the two files - the only field that differs is the artist name.

I guess this is ultimately an issue of jank on the Azlyrics side, since the site directory has separate listings for 'Liz Phair' and 'Phair, Liz' in their artist directory (which both lead to the same url, https://www.azlyrics.com/p/phair.html). But it would be nice if the scraping pipeline handled deduplication.

I did a quick analysis and found 6,513 total rows with duplicate song urls.

Would be very cool to integrate Tor and more techniques for making the requests stronger and avoiding 403 errors. This issue will be worked in tor_integration branch.