Infinite loop when generate lattice reference about pseudo-ref HOT 5 CLOSED

Hi ASLONGAS, thank you for your interest.

I just ran those references, it seems that it's not an infinite loop. The codes runs extremely slow because the lattice is very big and it takes a long time to do DFS. This lattice graph basically have 3,953,664 paths according to your parameter settings.

You can count the number of paths of a lattice from the following python function (I may add this to this repository recently):

def count_path(G):
    g_sum = {}
    for n in nx.topological_sort(G):
        if n == 0:
            g_sum[n] = 1
            continue
        g_sum[n] = sum([g_sum[u] for u, v in G.in_edges(n)])
    return g_sum[-1]

Please refer to the "Generation Speed" subsection in README about the solutions dealing with big lattices. Compared to the image captioning dataset, it also takes me a lot more time to generate pseudo-refs for NIST because the references are much longer than MSCOCO (please refer to Fig. 6 of the paper). I do dynamic threshold and multi-threading to get the final lattices (takes about 2 days for the 2002-2004 training set used in the paper).

from pseudo-ref.

Thank you for your reply, it works.
Thanks.

from pseudo-ref.

Hi renj! Sorry for bothering you again.

I met a case where there is a very large amount of paths for an instance in NIST2002 dataset when generating pseudo-refs. It has 35,675,280,506,880 paths calculated by your provided code. The instance is shown as follow：

允许 球员 在 进球 后 脱 掉 球衣 以 示 庆祝 , 但是 前提 是 不 能 影响 比赛 正常 进行 ; 鉴于 有 球员 在 贴身 背心 上 写 一些 容易 引发 矛盾 的 话 , 国际 足联 规定 , 从 今年 7月 1日 开始 , 球员 的 背心 上 不得 书写 任何 文字 ; 国际 足联 允许 球员 的 运动服 上 出现 广告 , 但是 必须 在 运动服 的 前面 。
players are now allowed to take off their sports clothing after scoring as a way of celebration provided that this would not affect the normal process of the game . as some players may write some potentially provocative messages on their inner vests , fifa prescribes that starting from july 1 of this year , players ' vests must not have any text . fifa allows advertisements on the player 's sports clothing , but they can only appear on the front of the clothing .
fifa permits football players to take off their uniform to celebrate a score . the prerequisite is that such act will not interfere the normal proceeding of the game . however , fifa is seeing that vests or underclothes of players sometimes have words on them , which could trigger contradiction . fifa now stipulates that the vests of players must not have any writing on them . the new rule is to be effective from july 1 . fifa will also permit advertising on the sportswear of players . however , the advertisement must be on the front part of the sportswear .
players will be allowed to take off their team shirts in celebration of goals , to such extent as may not affect the normal going of the match . given that some players put down words on their inner vests that are likely to invoke controversies , no writings will be allowed on the inner vests , starting from july 1 this year . fifa allowed advertisements on the team shirts , but only on their front side .
the players are allowed to take off their shirts to celeberate goals , on condition that the normal process of the game should not be disrupted . since some players write some controversial words on the backs of the inner vests , fifa hereby stipulates that no words could be written on these vests . fifa allows ads on the shirts , but they should be on the back .
[Quoted from NIST2002 the 14th sentence]

Since you've generated pseudo-refs from NIST02-06, I would like to know that whether there are some tricks for the generation? Or if I did something wrong?

Also, I used the setting as before:

I didn't delete punctuation in sentences[tokens]
Then I run this:
python lattice.py -order_method hard -align_method hard -dataset data/dataset.json -minus 0.5

In addition, could you please tell me the configurations of your machine (cpu, ram, etc.) ?

Hope for your kindly reply. Thank you so much!

from pseudo-ref.

Please check the paragraph next to image 6.

To generate enough references for the following experiments, we set an initial global penalty as 0.9 and gradually decrease it by 0.05 until we collect no less than 100 references.

I do the generation on a 1080Ti GPU server and use multi-threading. If I remember it right, the server has about 256G ram.

from pseudo-ref.

Thank for your kind and great help~

from pseudo-ref.