bdkiran / lyragen Goto Github PK

Add logic to the application that

• Fetches all songs from az lyrics from a certain artist

• Fetches all lyrics from the artist

We need this to have clean data regarding Artist Name and Song Name(currently all lowercase)

To resolve an issue of duplicate lyrics being returned, the best solution is to return store the duplicate lyrics in elastic as a single document. This document will represent if there are multiple lyrics by representing the multiple positions that they occur within the song.

• Duplicate Song lyrics are detected

• They are stored in a data structure that tracks the occurrence by line number

• These data structures are appropriately transformed to be stored in elasticsearch

Convert songs meta data storage to PostgreSQL DB instead of files.
Table songs
-song_id
-title
-artist
-album
-year
-lyrics_fetched

• Drop table

• Script completed, needs to be rerun to on both folders

Chars that are scraped from A-Z Lyrics that contain " ' " are being transformed to odd characters.
When scraping, these chars need to be cleaned then uploaded to elasticsearch

Using the json data structure dependency. Be able to update the database with multiple songs

• Add retry mechanism to accommodate when the API fails

• Update json document to reflect that a song was successfully updated

• Implement proper mechanisms to avoid rate limiting/bans from AZ lyrics

Update readme to:

• User proper markdown

• Explain what the project is for

• Instructions for downloading and dependencies

Song Lyrics should be able to be fetched from DB instead of from files.

To avoid Rate limiting issues:

• Either limit by amount

• Limit by artist

Have one JSON document used to feed in new songs

• Contains information Artist and Song Title

• Contain information if the track was successfully uploaded to db