Add WikilinkNN Unwanted Links about wikirec HOT 3 OPEN

@victle, I figure we can switch the discussion for #36 and #47 over here. Am currently re-running everything to make a new up to date enwiki_books.ndjson, books_embedding_model.h5 (renamed it to make it explicit), and will update the results for rec_books.ipynb. I guess this issue would be the next thing to do on this model, but then am open for suggestions :)

Again, sorry that I was awol for a bit there. Feel free to let me know what you have interest in doing, and of course no stress if you're busy. I'll be fully switching over to another project tomorrow, but would be happy to discuss all this with you still 😊

from wikirec.

This feels super specific to each kind of category, so I really wonder what the best approach is for removing some of these more common terms. I guess one brute force approach is just doing it empirically for different categories? Like, when generating an embedding for movies, comb through some of the top most frequent items and choose for removal? But that does seem like a lot of work 😅

Maybe another easy way is to filter certain wikilinks using regex? It seems to me that an easy one would be removing any links with "wikipedia" or "wikiproject" in them. I wonder how the performance of the model recommendations gets affected by having these extra links in them? It could be harmless for certain categories, but I haven't tested enough to know 🤷‍♂️

And no worries about being away! Everyone's got a life outside GitHub 😆 . My time is a bit limited moving forward, but I'd be happy to do small tasks here and there (e.g., perhaps picking out some unwanted links for movies or something?).

from wikirec.

I agree that those that have "wikipedia" and especially "wikiproject" should likely be removed. Maybe not the former as there could be things like categories that would be important, but then I'm not sure :) Maybe for major things it'd be best to select them, and for others we could just select those that are present in > X% to be removed.

By the sounds of it this is something that's later on the horizon for both of us though 😄 This especially given your point in the second paragraph above - how much is all this really going to help? :) Let's get back to it another time 😊

Feel free to check base here when you have a bit more time, or also let me know if you have something open source you'd want some help on!

from wikirec.