zhezhaoa / ngram2vec Goto Github PK

I'm getting this divide by zero error during the evaluation of every embedding when doing ngram_ngram.

Counts2glove finished
seen/total: 1/203
testsets/similarity/ws353_similarity.txt: nan
Traceback (most recent call last):
  File "/home/mpickard/Projects/ngram2vec/ngram2vec/ngram2vec/analogy_eval.py", line 102, in <module>
    main()
  File "/home/mpickard/Projects/ngram2vec/ngram2vec/ngram2vec/analogy_eval.py", line 75, in main
    accuracy_add = float(correct_add) / seen
ZeroDivisionError: float division by zero

I'm using Python 3. Any ideas on why seen is zero in the analogy.eval.py code? And why sim_actual and sim_expected end up with a correlation of zero in the similarity_eval.py code? I was trying the code on a small corpus.

Example use cases中有预处理过（分好词的）的中文维基语料，由于想尝试统一分词器以便处理OOV的问题，可以问下所用的分词器是什么吗？

hi，我想问下，预测里的两个指标是什么？是否越高越好呢？

谢谢开发者的分享！！！
我想实现bigram，请问实现bigram的源码在哪个文件夹呢？

Can someone verify that I'm thinking correctly about how to generate 6-grams. In the ngram_example.sh script, I simply change the "order" and "output_order" values to 6, correct?

...
python ngram2vec/corpus2vocab.py --corpus_file ${corpus} --vocab_file ${output_path}/vocab --memory_size ${memory_size} --feature ngram --order 6
python ngram2vec/corpus2pairs.py --corpus_file ${corpus} --pairs_file ${output_path}/pairs --vocab_file ${output_path}/vocab --processes_num ${cpus_num} --cooccur ngram_ngram --input_order 1 --output_order 6
...

应该如何安装representations.matrix_serializer，我的pip找不到相应的版本？

您好，首先感谢贵组的开源，我想咨询下在最后生成向量里面的input_vector_file是

什么文件？vocab，counts这些文件都是预先生成的，但是关于这个文件比较模糊，非常感谢！

后面的也走不了了
Pairs2sgns
Traceback (most recent call last):
File "ngram2vec/pairs2sgns.py", line 53, in
main()
File "ngram2vec/pairs2sgns.py", line 47, in main
return_code = subprocess.call(command)
File "/home/anaconda3/lib/python3.6/subprocess.py", line 267, in call
with Popen(*popenargs, **kwargs) as p:
File "/home/anaconda3/lib/python3.6/subprocess.py", line 709, in init
restore_signals, start_new_session)
File "/home/anaconda3/lib/python3.6/subprocess.py", line 1344, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory:'./word2vec/word2vec': './word2vec/word2vec'
Traceback (most recent call last):
File "ngram2vec/similarity_eval.py", line 71, in
main()
File "ngram2vec/similarity_eval.py", line 41, in main
matrix, vocab, _ = load_dense(args.input_vector_file)
File "/home/yongzhuo/ngram2vec/ngram2vec/utils/matrix.py", line 23, in load_dense
with codecs.open(path, "r", "utf-8") as f:
File "/home/anaconda3/lib/python3.6/codecs.py", line 897, in open
file = builtins.open(filename, mode, buffering)
FileNotFoundError: [Errno 2] No such file or directory: 'outputs/wikipedia/word_word/sgns/sgns.input'

I am confused about the sub-sampler in corpus2pairs. I think 1 - sqrt(subsample / count) should be replaced with 1 - sqrt(subsample / (count / total_word_count_in_vocab)).

ps. I might misunderstand your implementation, and in actual implementation of original word2vec.c ，the subsample probability equals 1 - (sqrt(subsample / (count / total_word_count_in_vocab)) + subsample / (count / total_word_count_in_vocab) ).

你好，最近在用ngram2vec工具，有点困惑，要得到word+character+ngram这种context Features，我的语料要怎么处理呢？分词还是分字？
如果是分词的话，脚本里要怎么传参数才能得到character特征呢？ 我在代码里看没有找到这部分内容

Hi，首先感谢贵组将开源embedding工具ngram2vec。我在用ngram2vec时出错，想请教一下。

uni-bi.sh 执行出错 ./word2vecf/word2vecf: cannot execute binary file

Mac OS 10.13.6

查阅资料说是 在Linux下的执行文件在MacOS上无法运行。
https://superuser.com/questions/724301/how-to-solve-bash-cannot-execute-binary-file

According to your file output, this program is for GNU/Linux. I know this because:

The file b1 is in the ELF (Extensible and Linkable Format) format, while Mac OS X uses the Mach-O format for binaries;

file recognizes this file is for GNU/Linux 2.6.18, meaning it'll work on most modern Linux distributions.

To solve your problem, you must either run this problem within a Linux distribution, recompile the program, or get the Mac OS X version of this program.

想请问uni_bi.sh 计算SGNS的word2vecf/word2vecf是否为word2vec/makefile执行编译后生成的二进制文件呢？ 可否用macOS重新编译呢？
······································
另外，为什么demo_simplified.sh 不需要执行word2vecf二进制文件，执行word2vecf.py也可以计算SGNS，word2vecf二进制执行文件 和 word2vecf.py是什么关系呢？

In line2pairs, line 53 and 56, it checks whether the input ngram and the output ngram overlap (completely or partially).
For complete overlap, I reckon you have to check if the first token of each ngram is the same and if both ngrams are the same length. However, this last check is performed with the input_order and output_order variables, that don't represent those particular ngrams' length but the maximum ngram's length to search for in the line. For example, if you have input_order = 1, output_order = 2 and overlap = True, you will never pass the input_order = output_order check and therefore you will eventually get ngrams paired with themselves.
The same thing happens with the partial overlap check.

Shouldn't line 53 be
if i == l and j == k:
instead of
if i == l and input_order == output_order:

And line 56
if len(set(range(i, i + j)) & set(range(l, l + k))) > 0:
instead of
if len(set(range(i, i + j)) & set(range(l, l + k))) > 0:

您好，感谢贵组开源了ngram2vec，这对我的帮助很大，谢谢～

如果使用预训练SGNS词向量再根据自己的语料库fine-tune词向量，是使用预训练词向量初始化embedding，是否需要修改代码呢，是从word2vecf下手吗？

期待您的回复。

Hi，感谢贵组开源了ngram2vec工具～

我从CA8论文中了解到Context中添加ngram+char+word的embedding在中文语料中效果很好。
我想要训练自己语料库的SGNS ，context为ngram+char+word，ngrm2vec工具包已经实现了context中添加了ngram，请问如果要在context中添加char特征需要做哪些工作呢？

期待您的回复。

Thank you for your great work, but how can I get the PPMI of 'bi_bi' type?

您好，请问中文语料的格式是什么样的？百度网盘的连接现在不能下载了。
我用自己的语料训练会报错？不知道能够提供一下中文语料的格式，谢谢！

首先感谢大佬的开源，我想问一下为什么新版中pairs2sgns.py或者counts2glove没有提供一些训练参数接口，例如window，min_count。都是使用默认的？如果想要调整需要修改c代码，或者不使用python，直接c版本运行？最后再次谢谢大佬。

I'm trying to create uni_bi.sh with Chinese/utf8 word seg file, however always got following error.
any idea ?

==========
Traceback (most recent call last):
File "ngram2vec/pairs2counts.py", line 109, in
main()
File "ngram2vec/pairs2counts.py", line 88, in main
counts_file.write(str(old[0]) + " " + str(w) + " " + str(old[1][w]) + "\n")
TypeError: write() argument 1 must be unicode, not str

Hello, i am using your paper for my own thesis research. Looking at the workflow.jpg diagram, the arrows are a bit confusing. I am trying to use the Skip-Gram ngram-ngram method. From what i understand, it seems that i have to go through the steps corpus2vocab -> corpus2pairs -> paris2sgns. But paris2sgns requires an "--input_vector_file" argument. I dont know what that is and the steps didnt generate one. I assume its the resulting word embeddings vectors in a file, but if i have that, then i wouldnt be using the tool. Do i have to run the original word2vec SG method and save a .vec model and use it here? I read the research paper and didnt find an answer to this either. I also tried pairs2vocab, but it also doesnt generate the input vector file.

A separate issue is with the corpus2pairs; it generates 4 different .txt files (pairs.txt_0, pairs.txt_1, pairs.txt_2, pairs.txt_3), when i give the argument "--pairs_file ./pairs.txt". Then later do i have to run paris2sgns for all pairs files? Do i generate different output vector files for each? Do the vector files get overwritten or appended to?

如果要追加训练，要怎么操作？