andras7 / word2vec-pytorch Goto Github PK
View Code? Open in Web Editor NEWExtremely simple and fast word2vec implementation with Negative Sampling + Sub-sampling
Extremely simple and fast word2vec implementation with Negative Sampling + Sub-sampling
hi!
pow_frequency = np.array(list(self.word_frequency.values())) ** 0.5
should not be
pow_frequency = np.array(list(self.word_frequency.values())) ** 0.75
It is not an issue. I just want to ask why do you use running_loss = running_loss*0.9 + loss.item()*0.1 for monitoring the loss during training?
Do you have any special reason for this?
Isnt it conventional to monitor the average loss after each epoch (in this case, after each iteration)?
Thanks for sharing your implementation of Word2Vec!
One question, where did you get this formula for the implementation of subsampling? I don't seem to see it in the paper or the C-code.
word2vec-pytorch/data_reader.py
Line 55 in 5281d15
Hi @Andras7
I reorganized your code a little bit to make it easily installable with pip.. You can install it with:
pip install git+https://github.com/marta-sd/word2vec-pytorch.git
You can take a look at my version here
If you are interested in this kind of contribution I'd be happy to create a PR
Best
MSD
Hello!Thanks for your code! Have you observed the loss? I had loaded the code and executed it. However, the loss didn't seems to be convergent. It descends rapidly at first, but with the increase of the epoch, it looks like a cosine function and the amplitude increases too.
Hi, @Andras7.
Thank you for your contribution.
I have one question about class Word2vecDataset(Dataset).
In getitem(self, idx), is window_size correct?
return [(u, v, self.data.getNegatives(v, 5)) for i, u in enumerate(word_ids) for j, v in
enumerate(word_ids[max(i - boundary, 0):i + boundary]) if u != v]
I think this code returns wrong windows (word_ids[max(i - boundary, 0):i + boundary])) and following code (word_ids[max(i - boundary, 0):i + boundary+1])) may be correct.
return [(u, v, self.data.getNegatives(v, 5)) for i, u in enumerate(word_ids) for j, v in
enumerate(word_ids[max(i - boundary, 0):i + boundary+1]) if u != v]
If it is not wrong, I'm sorry for that.
In addition to this, it may not be important and I don't have confidence.
if u != v
needs to change if i != j
.
Why add (t/f) in this formula for discards:
t = 0.0001
f = np.array(list(self.word_frequency.values())) / self.token_count
self.discards = np.sqrt(t / f) + (t / f)
word2vec-pytorch/word2vec/data_reader.py
Line 102 in 36b93a5
I think i + boundary
should include a + 1
to make it inclusive, otherwise the right context takes 1 token less in the resulting skipgrams.
Hello Andras7,
Thank you for this fast and effective implementation of word2vec. We have forked your repository and added some augmentations for a research project, and would like to properly acknowledge your github repository in the final research paper. What name and/or website would you like us to reference when acknowledging our use of the code?
(If you prefer not to disclose your answer publicly, feel free to email me directly at [email protected].)
Many thanks!
Nancy Fulda
Assistant Professor
Brigham Young University
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.