jeffreygaydos / bandcamp-rss-feeds Goto Github PK

Bandcamp allows artists to redirect users to their most recent release when they navigate to their artist link without the /music postfix. I believe the /music version of artist links brings up the discography regardless of if an artist has opted in to the redirection.

When creating the release updates, we sanitized the /music postfix in order to actually see the artists discography. We need to do the same for follow links on output.

Even when we make our first feature, we will need it to show for multiple users to be truly feature pair with the original system. Augment whatever RSS aggregator we start with to show 1 RSS feed element for each user (I think Monito will handle this the best and we don't plan on having more than a few users on this thing).

Here's what we need:

• Use a SSPF to store which users are currently "enrolled" in the bandcamp updates system. For now, if the user is present there, they will get all updates.
• Read from the SSPF to determine which users to run for, and run for those users daily.
• In the aggregation step, do so for each user, and put each user's summary as a separate entry in the RSS feed
  • Before doing so, check that the user had any updates to display
    • This can be denoted directly by the runners or by the commit dates on the SSFs that the aggregator looks at
  • For the users that had updates, add an RSS "article" for them
  • Monito integration should already be there

This thing is going to be a little too complicated to be merged with the first set of parsing that we do, so let's focus on this one individually.

here's what needs done:

• Look at a list of SSFs parsed out of various bandcamp websites
  • Based on the name of the SSF and the last commit date of the SSF, add values to a part of the final aggregated RSS feed
    • This might need to be done in GHA if we can't access commit history from a file. Otherwise, we can add a timestamp to the top of each SSF (leaning towards the timestamp), but only when we actually make an update to the file
    • If there is at least 1 SSF with an update date that is recent, we know that SSF has new data.
  • Ensure that this happens after the other SSFs have been written to by the daily udpate
    • Keep it all in one GHA that is blocked on the previous steps completing
• Take the SSFs that have new data in them, string interpolate the data into a categorized RSS feed article block
• Put that article block into an RSS feed looking wrapper
• Commit and push changes to the RSS feed file ONLY if you found an SSF that changed
• Integrate with Monito RSS and the #bandcamp-updates channel

Releases from years ago are appearing in the daily updates for an unknown reason. At the same time, existing releases are being removed, meaning that those removed releases will appear again if we ever find them in the future.

Bandcamp may have started paginating artist home pages. I recall being surprised that it was not already paginated when this was created.

Consider attempting to pull data directly from their API instead of using homepage HTML

Maybe because I made some changes and didn't properly sync the SSFs to the rss branch (making this issue temporary), but occasionally, artists that have been followed by users in the last week or so have all of their current releases dumped in the feed roughly 5 days after. Investigate why this is happening and attempt to fix it.

A reach goal for sure, but let's add recommendations that bandcamp gives you based on what you've clicked on in their site, just like the emails.

Here's what needs done:

• Parse the bandcamp feed section for a user
  • Look only for those posts that have the title "Suggested in {genre}" and grab the release link from the post
  • Store these posts in an SSF if they are new. If they are not new (according to what is already in the SSF), don't change the SSF
  • Be sure to preload the SSF if you want to ignore the current posts
• Integrate with the RSS Aggregator

Blocked on figuring out how to even access the feed without logging in...

This should be a victory lap. Just make a small script that appends another rss block at the end of the existing feed, and call it from a GHA. See if this media link will transfer to discord properly: https://tenor.com/view/ohhp-gif-4817530786571504521

Bandcamp has a super sick API that you can just call and get every little piece of data you could possibly want! WOW! Let's use this thing and stop messing around with urls when all we want is the data!

This is necessary for getting "all" followers, "all" wishlist updates and "all" collection updates.

Here's what we need to do:

• Use Beautiful soup to send a post request (or use another library if that doesn't pan out)
• Parse the data returned by bandcamp to extract particular bits of data in a generic way for the 3 parsers
• Research into if we can do the same thing for particular artists that a user follows

NOTE: artist metadata is not really in a public facing API as far as I can tell, so you might need to rely on the website view for that one.

Now that we have the followers updating, we can take that information and look for new releases from those followed artists.

Here's what we need to do:

• Take the SSPF from the followers update RSS thing and loop through each artist link
  • In each artist link, parse the page to find links to releases, and add those releases to an SSF
    • The first run should return all releases from all followed artists, which is fine
  • Before adding the link to the SSF, check to see if the link already exists in the SSF, and if it does, don't change the SSF. If the link is new and this is a new release. Store the link in memory and continue until reading through all artists, then update the SSF
  • The SSF will be for all artists with no distinction between them, but if we detect 2 releases from the same artist, they should be listed next to each other in the SSF
• Aggregate the SSF into a new section of the main RSS feed, outputting just the links and relying on discord's embed magic
• Integrate with MonitoRSS to get the updates posted in the #bandcamp-updates channel

Ok now that we have approximate feature parity with the original RSS feeds, let's add some stuff, specifically the new artists that a user is following. We don't really care about this, but it will seriously help us get the new releases pipeline going, with considerable code overlap.

Here's what needs done:

• Parse out the following page for a user
  • Get each "new" artist's link and add it to an SSPF (super simple persistent file) named as {bandcamp username}_following_persist. Also store new artists for that run in the SSF that will be used by the RSS aggregator
    • When the SSPF is empty, we will of course "ingest" all of the followers, but this will likely only happen once (neat!)
    • When the SSPF is not empty, it will add the ones that are new to the SSPF, and an SSF that will be used by the RSS aggregator
• Tweak the RSS aggregator to accept and merge follow updates and wishlist updates
• Integrate with MonitoRSS to post in the #bandcamp-updates channel

We use a 3rd party to generate RSS feeds for bandcamp currently (this thing). Let's migrate off of that and make our own RSS feeds using github actions and the raw view of a code file!

This includes the following features:

• Parse the wishlist page of a bandcamp user's profile
  • Extract the link from the wishlist item, discord will put a nice embed so no other information should be needed
  • Grab as many links as needed until you see the most recent wishlist item that was present in a previous notification
    • (this limits the number of udpates we can give in 1 day to 20 releases, hoepfully that's enough but we can always add additionally functionality to click the "load more" button)
  • If there are no true updates to make, don't change the RSS feed at all so we don't get a repeat or empty message from Monito
    • For later features, we will need to clear out the "wishlist udpates" if there is a change in another feed since we don't want the last item wishlisted to show up repeatedly
  • Put the link in a SSF (super simple file), probably just a list of URLs separated by newlines
• Integrate with the RSS Aggregator

This could be as simple as storing the top 20 wishlist items in the SSF, making it more of an SSPF, but then all of them would be persistent and we could just call them SSFs. Essentially, we would check for new items and if we found them, we would update with the current top 20. If we found no new items we would leave the top 20 already in the file. When we need to aggregate them, we only want the new ones, which we could prefix with NEW: or something

Here's what needs done:

• When you find a repeat wishlist item (i.e. already exists in the SSF), don't end, keep looking and store the top 20 in the SSF without any prefixes
• When you find a new wishlist item, keep looking and store it in the file as NEW: {link to release}
• Update the RSS aggregator to only look at entries that have the NEW : prefix

Let's add a thing that let's other know when a user has bought something using the collections page on bandcamp.

Here's what needs done:

• Parse the collections page to get any new collection items for this user
  • If you find a new collection item (according to the SSF), store it in memory and replace the SSF at the end
  • The first time you don't find a new collection item, stop parsing and don't update the SSF
  • We are opting to not attempt to click the view more button here because it is likely that the user will not buy/acquire 20+ items in 1 day (that's $200 if they are acquiring real albums)
  • Again, on the first run, we will likely see all of the items, which would then update the SSF
    • Preload the SSF if you want to avoid this
• Integrate the collection SSF with the RSS aggregator