Would you please share resources used in your implementation like “ukwac.model”，“ukwac_id2word.pkl“，“ukwac_word2id.pkl” about comhyper HOT 4 CLOSED

The ukwac.model is trained using gensim word2vec tools based on the corpus ukWac. The official download site is https://wacky.sslmit.unibo.it/doku.php?id=corpora since we are not supposed to distribute it.

from comhyper.

The ukwac.model is trained using gensim word2vec tools based on the corpus ukWac. The official download site is https://wacky.sslmit.unibo.it/doku.php?id=corpora since we are not supposed to distribute it.

Thanks to your replay.

• I've already downloaded ukwac（non dependency-parsed, non-pos tagged, plain text version） according to your description. Is this the version you utilized?

• Besides, I encountered "UnicodeDecodeError" when I tried to use gensim with "word2vec.Word2Vec(LineSentence('ComHyper main/ukwac_preproc/cleaned_pre.pos.corpus'),size=300,min_count=1,sg=1)". Have you encountered similar problems? I would be very grateful if you can share the script about training word embeddings, considering hyper parameters like "min-count" will affect the final result.

from comhyper.

Generally I use the following code to train the gensim word2vec model as introduced in his official blog. https://rare-technologies.com/word2vec-tutorial/. The parameters are set default such as min_count = 5.

class MySentences(object):
    def __init__(self, dirname):
        self.dirname = dirname
 
    def __iter__(self):
        for fname in os.listdir(self.dirname):
            for line in open(os.path.join(self.dirname, fname)):
                yield line.split()
 
sentences = MySentences('/some/directory') # a memory-friendly iterator
model = gensim.models.Word2Vec(sentences)

from comhyper.

Generally I use the following code to train the gensim word2vec model as introduced in his official blog. https://rare-technologies.com/word2vec-tutorial/. The parameters are set default such as min_count = 5.

class MySentences(object):
    def __init__(self, dirname):
        self.dirname = dirname
 
    def __iter__(self):
        for fname in os.listdir(self.dirname):
            for line in open(os.path.join(self.dirname, fname)):
                yield line.split()
 
sentences = MySentences('/some/directory') # a memory-friendly iterator
model = gensim.models.Word2Vec(sentences)

Thanks a lot! Wish you a happy day.

from comhyper.