cranial-xix / generative-neural-parser Goto Github PK

I noticed you are using LSTM-minus for the inside span of C, which I' not sure is OK. The minus operation in the NN sometime causes issues which results high weights. (because large weights won't be penalized after minus) Is that possible to just use CIv ?

I assume you are using LN.

1. Generative-Neural-Parser/model.py

   Line 356 in 693b53d

   self.relu = nn.ReLU()

   Any particular reasons for relu? I know it is a popular choice but it won't bound the outputs. This happens to me before where relu is no better than tanh or sigmoid. It might be interesting

2. Generative-Neural-Parser/model.py

   Lines 378 to 381 in 693b53d

   	B_h1_init = self.initrange(B_in, d_B)
   	B_h2_init = self.initrange(d_B, B_out)
   	self.B_h1.weight.data.uniform_(-B_h1_init, B_h1_init)
   	self.B_h2.weight.data.uniform_(-B_h2_init, B_h2_init)

3. Generative-Neural-Parser/model.py

   Line 370 in 693b53d

   zeta = 0.4

   , what is zeta?
   You're basically implementing Xavier Initialization right? I remember the default init of Pytoch it already that. Is that true?

4. The implementation here is not a good practice:

   Generative-Neural-Parser/model.py

   Lines 443 to 550 in 693b53d

   	PP = self.nt_emb(Variable(torch.from_numpy(P_P).cuda()))
   	PI = torch.index_select(alpha, 0, Variable(torch.from_numpy(P_i).cuda()))
   	UA = self.nt_emb(Variable(torch.from_numpy(U_A).cuda()))
   	UI = torch.index_select(alpha, 0, Variable(torch.from_numpy(U_i).cuda()))
   	BA = self.nt_emb(Variable(torch.from_numpy(B_A).cuda()))
   	BI = torch.index_select(alpha, 0, Variable(torch.from_numpy(B_i).cuda()))
   	CA = self.nt_emb(Variable(torch.from_numpy(C_A).cuda()))
   	CB = self.nt_emb(Variable(torch.from_numpy(C_B).cuda()))
   	CI = torch.index_select(alpha, 0, Variable(torch.from_numpy(C_i).cuda()))
   	CJ = torch.index_select(alpha, 0, Variable(torch.from_numpy(C_j).cuda()))
   	x2y = self.lsm(
   	self.B_h2(
   	self.relu(
   	self.B_h1(
   	torch.cat( (BA, BI), 1 )
   	)
   	)
   	)
   	).data.cpu().numpy()
   	xy2z = self.lsm(
   	self.C_h2(
   	self.relu(
   	self.C_h1(
   	torch.cat( (CA, CB, CI, CJ-CI), 1 )
   	)
   	)
   	)
   	).data.cpu().numpy()
   	x2u = self.lsm(
   	self.U_h2(
   	self.relu(
   	self.U_h1(
   	torch.cat( (UA, UI), 1 )
   	)
   	)
   	)
   	).data.cpu().numpy()
   	lex = self.lsm(
   	self.T_h2(
   	self.relu(
   	self.T_h1(
   	torch.cat( (PP, PI), 1 )
   	)
   	)
   	)
   	).data.cpu().numpy()
   	else:
   	PP = self.nt_emb(Variable(torch.from_numpy(P_P)))
   	PI = torch.index_select(alpha, 0, Variable(torch.from_numpy(P_i)))
   	UA = self.nt_emb(Variable(torch.from_numpy(U_A)))
   	UI = torch.index_select(alpha, 0, Variable(torch.from_numpy(U_i)))
   	BA = self.nt_emb(Variable(torch.from_numpy(B_A)))
   	BI = torch.index_select(alpha, 0, Variable(torch.from_numpy(B_i)))
   	CA = self.nt_emb(Variable(torch.from_numpy(C_A)))
   	CB = self.nt_emb(Variable(torch.from_numpy(C_B)))
   	CI = torch.index_select(alpha, 0, Variable(torch.from_numpy(C_i)))
   	CJ = torch.index_select(alpha, 0, Variable(torch.from_numpy(C_j)))
   	x2y = self.lsm(
   	self.B_h2(
   	self.relu(
   	self.B_h1(
   	torch.cat( (BA, BI), 1 )
   	)
   	)
   	)
   	).data.numpy()
   	xy2z = self.lsm(
   	self.C_h2(
   	self.relu(
   	self.C_h1(
   	torch.cat( (CA, CB, CI, CJ-CI), 1 )
   	)
   	)
   	)
   	).data.numpy()
   	x2u = self.lsm(
   	self.U_h2(
   	self.relu(
   	self.U_h1(
   	torch.cat( (UA, UI), 1 )
   	)
   	)
   	)
   	).data.numpy()
   	lex = self.lsm(
   	self.T_h2(
   	self.relu(
   	self.T_h1(
   	torch.cat( (PP, PI), 1 )
   	)
   	)
   	)
   	).data.numpy()

   , you might want wrapper of the Variable(..) with pays attention to the device.

5. How large is self.nnt? If it is large, is that possible to simplify things like.
   

   Generative-Neural-Parser/model.py

   Lines 592 to 594 in 693b53d

   	self.B_h1(
   	torch.cat( (AAv, BIv), 1 )
   	)

   
   ？self.nt_emb is a 1-hot embedding. So this is basically concatenating 1 row in the upper part of self.B_h1 with the multiplication output of the lower part.