HAKE-Large+iCAN? about hake-action HOT 3 CLOSED

To combine our HAKE-large with other instance-level HOI detection methods (like iCAN), there are two circumstances:

1. You are using the same object detection results as TIN and HAKE-large. For this, replace the score files as shown in the following code with corresponding result files.
   

   HAKE-Action/-Results/Generate_detection.py

   Lines 28 to 30 in ddce236

   	score_H = pickle.load(open('TIN/score_H.pkl', 'rb'))
   	score_O = pickle.load(open('TIN/score_O.pkl', 'rb'))
   	score_sp = pickle.load(open('TIN/score_sp.pkl', 'rb'))

   
   And if necessary, feel free to replace the scale factors and weights in the following code to fit the customized result.
   

   HAKE-Action/-Results/Generate_detection.py

   Line 42 in ddce236

   h_fac, o_fac, sp_fac, P_fac, A_fac, L_fac, hthresh, othresh, athresh, bthresh, P_weight, A_weight, L_weight = pickle.load(open('generation_args.pkl', 'rb'))

   
   And corresponding changes might be made to the following code.
   

   HAKE-Action/-Results/Generate_detection.py

   Lines 63 to 78 in ddce236

   	for hoi_index in range(x, y):
   	score_H[obj_index][:, hoi_index - x] /= h_fac[hoi_index]
   	score_O[obj_index][:, hoi_index - x] /= o_fac[hoi_index]
   	score_sp[obj_index][:, hoi_index - x] /= sp_fac[hoi_index]
   	score_P[obj_index][:, hoi_index - x] /= P_fac[hoi_index]
   	score_A[obj_index][:, hoi_index - x] /= A_fac[hoi_index]
   	score_L[obj_index][:, hoi_index - x] /= L_fac[hoi_index]
   	hod = getSigmoid(9, 1, 3, 0, hdet[obj_index].reshape(-1, 1)) * getSigmoid(9, 1, 3, 0, odet[obj_index].reshape(-1, 1))
   	sH = torch.sigmoid(torch.from_numpy(score_H[obj_index]).cuda()).cpu().numpy()
   	sO = torch.sigmoid(torch.from_numpy(score_O[obj_index]).cuda()).cpu().numpy()
   	ssp = torch.sigmoid(torch.from_numpy(score_sp[obj_index]).cuda()).cpu().numpy()
   	sP = torch.sigmoid(torch.from_numpy(score_P[obj_index]).cuda()).cpu().numpy() * P_weight
   	sA = torch.sigmoid(torch.from_numpy(score_A[obj_index]).cuda()).cpu().numpy() * A_weight
   	sL = torch.sigmoid(torch.from_numpy(score_L[obj_index]).cuda()).cpu().numpy() * L_weight
   	sHO = (((sH + sO) * ssp + sP + sA + sL) * score_I[im_index[keys[obj_index]], x:y]) * hod

2. You are using different object detection result from TIN and HAKE-large. Besides replacing the score files, the other files also needs re-generation. Note that pos.pkl and neg.pkl are the interactiveness prediction results from TIN model.
   

   HAKE-Action/-Results/Generate_detection.py

   Lines 28 to 40 in ddce236

   	score_H = pickle.load(open('TIN/score_H.pkl', 'rb'))
   	score_O = pickle.load(open('TIN/score_O.pkl', 'rb'))
   	score_sp = pickle.load(open('TIN/score_sp.pkl', 'rb'))
   	hdet = pickle.load(open('TIN/hdet.pkl', 'rb'))
   	odet = pickle.load(open('TIN/odet.pkl', 'rb'))
   	keys = pickle.load(open('TIN/keys.pkl', 'rb'))
   	pos = pickle.load(open('TIN/pos.pkl', 'rb'))
   	neg = pickle.load(open('TIN/neg.pkl', 'rb'))
   	bboxes = pickle.load(open('TIN/bboxes.pkl', 'rb'))
   	score_P = pickle.load(open(args.model + '/scores_P.pkl', 'rb'))
   	score_A = pickle.load(open(args.model + '/scores_A.pkl', 'rb'))
   	score_L = pickle.load(open(args.model + '/scores_L.pkl', 'rb'))

   
   If necessary, the scale factors, weights and code could also be changed following the above description.

And we will consider making our iCAN+HAKE-Large result public, while for better performance, TIN+HAKE-Large is still preferred.

from hake-action.

Thank you for the thorough response. As for making the iCAN+HAKE-Large code public, I think this would be a nice addition for people who want to tradeoff low complexity and low compute for some performance.

from hake-action.

Thanks for your advice! We will work on it.

from hake-action.