kimhc6028 / relational-networks Goto Github PK

Very nice work!
I suggest saving the pickle file for the dataset with protocol=2. This is faster in loading and saving and also uses 1.4GB instead of 5GB.

I do not understand well how you pair the objects. My question is if each pair of objects is feed as input once or twice. If it is fed once then how do you choose the order? to be independent of the order each pair should be fed twice, right? Otherwise there is some break of symmetry in the input pair. I could not understand this from the original article, and the code is still obscure to me to understand this. thanks!
UPDATE: I think I got it know, you input all pairs twice (in different order each time), and also each object with itself, thus n^2 instead of n(n-1)/2, Is this correct?

Hi, is there any public AMI on amazon so that we can try the code ? I cannot run your code a couple of different pytorch AMIs since I got a couple of errors inside the main.py file.

I am implementing something using CLEVR dataset and is using your repository for reference purpose.

But everytime I am executing the script, I am getting the error 'CNN_MLP' object has no attribute 'train'. In main.py right after the function 'train' starts, it executes model.train(). Somehow I cannot go beyond this point.

Could somebody help me to understand if I am doing something wrong?

EDIT: Sorry - this is my mistake. I was confused because apparently in numpy.random.randint the "high" is exclusive, whereas the python random.randint isn't. You can delete this comment.

In the model, there is no inclusion of an LSTM. How is the question then encoded? @gngdb Do you have any idea about this?

Say, if in the training data the RN saw many green circles, and many yellow triangles, but it never saw green triangles or yellow circles, would the RN perform well if asked a question about those shape-color combinations it never saw in the training data? In other words, can the RN learn the abstract concept of "color", the abstract concept of "shape" and generalize those concepts to understand new questions involving novel color-shape combinations?

How would one preprocess the bAbI tasks text into supposed objects and relations for loading into this training model?

The original paper does not seem entirely clear to me:

Dealing with natural language For the bAbI suite of tasks the natural language inputs must be transformed into a set of objects. This is a distinctly di↵erent requirement from visual QA, where objects were defined as spatially distinct regions in convolved feature maps. So, we first identified up to 20 sentences in the support set that were immediately prior to the probe question. Then, we tagged these sentences with labels indicating their relative position in the support set, and processed each sentence word-by-word with an LSTM (with the same LSTM acting on each sentence independently).

Hello sir,
I am reading the paper and your code. However there is one thing I dont understand from your implementation.
In the paper, at the Section 4 (Dealing with pixels), the author said that: "So after convolving the image, each of the d2 k-dimensional cells in the d x d feature maps was tagged with an arbitrary coordinate ...". So my question is in your code, which part is referring to this and would you please explain more about it since it is kind a difficult for me to understand it. Thanks !

We used vulture to search for unused code in your project. You can find the report below. It would be great if you could give us feedback about which items are actually used or unused. This would allow us to improve vulture and ideally it also helps you to remove obsolete code or even find typos and bugs.

Command

vulture relational-networks

Raw results

relational-networks/main.py:43: Unused variable 'kwargs'
relational-networks/main.py:138: Unused variable 's_datasets'
relational-networks/model.py:34: Unused function 'forward'
relational-networks/sort_of_clevr_generator.py:11: Unused variable 'question_size'
relational-networks/sort_of_clevr_generator.py:13: Unused variable 'answer_size'
relational-networks/translator.py:2: Unused function 'translate'

There might be false positives, which can be prevented by adding them to a
whitelist file. You may find more info here

Regards,
vulture team

Hi, the code is almost completely self-explanatory. However, I couldn't understand this part. Could you explain there ? Why you're creating coord_tensor and np_coord_tensor and what is the number 25 there ? I also would like to hear about lines 48-49-50.
Edit:
I have also another question about the implementation. Eventhough it is completely differ'rent, I don't want to open one more issue :)
I guess translator.py file and function inside of it isn't used anywhere in the code. What was the aim of that file ?

I have trained RN for 10 epochs. The final test set accuracy is 73% for relational question and 72% for non-relational question. It seems that there is no significant improvement for relational questions.

Hi, how did you select the questions generated by sort_of_clevr.py? Is there a standard specification anywhere?

Why all of your variables here and here's requires_grad argumand set to False ? You set requires_grad parameter of coord_tensor variable to false and then you concatenate it with the output of cnn (defined as x),whose requires_grad arguman is true by default I guess, at line 78. In this case what is the requires_grad parameter of concatenated variable (output of line 78 which is also x_flat)?

I wanted to report that I didn't manage to replicate the results in the paper or in the repo.
Relation accuracy: 80%
Non-Relation accuracy: 93%

trained for 20 epochs, with all default arguments

Regarding the training procedure for the entire CLEVR, how did you manage to train pixel and state description stages? i.e., did you train end-to-end the whole system (LSTMs, ConvInputModel, and RN)? Or was it by stages?

Another question off the topic: What is the purpose of coord_oi and coord_oj

Thank you! Great implementation by the way.

Hi, I'm trying to find a source for the Sort-of-CLEVR Dataset. The provided code in this repository seems to be what I'm looking for, but I need help understanding how to set it up for training, validation, and testing. Could you provide a brief example of how the included code could be used to generate a training, validation, and testing set, and then from this, iterate through these datasets in batches of chosen size?

Hi, thanks for your work and sharing of the code!!!
I have on question on data generation part,
I know the questions and answers are represented in one-hot vectors
where
questions = 2 x (6 for one-hot vector of color), 3 for question type(binary, ternary, norel), 3 for question subtype
answers = yes, no, rectangle, circle, r, g, b, o, k, y

My question is why you use count+4 in here bianry question-subtype 3, which is as follows:

  elif subtype == 2:
      """count->1~6"""
      my_obj = objects[color][2]
      count = -1
      for obj in objects:
          if obj[2] == my_obj:
              count +=1 
      answer = count+4

As I understand, the count is already the number of Number of objects which have the same shape with the certain colored object.
The +4 in [ yes, no, rectangle, circle, r, g, b, o, k, y] means the colors?

Any help would be appreciated and thanks for your time

Hello, this might be a stupid question but I have not seen any use of self like:
output = self(input_img, input_qst)
in the definition of BasicModel before. From later lines I know output must be a tensor but I cannot understand how it processes input_img and input_qst. If this is simply a Python or Pytorch question, not relevant to your code, could you please perhaps let me know where I can find relevant answers? Thanks!

Hi, in line 21 I had a hard time understanding the calculation
(24+2)*2+11
The input of the linear layer should be two objects each encoded as a vector size of 24.
Where did the +2 and +11 came from?

Thanks!

Never mind.

Train Epoch: 18 [193280/196000 (99%)] Relations accuracy: 95% | Non-relations accuracy: 100%
Train Epoch: 18 [194560/196000 (99%)] Relations accuracy: 86% | Non-relations accuracy: 100%
Train Epoch: 18 [195840/196000 (100%)] Relations accuracy: 89% | Non-relations accuracy: 100%

Test set: Relation accuracy: 89% | Non-relation accuracy: 100%
Train Epoch: 19 [192000/196000 (98%)] Relations accuracy: 94% | Non-relations accuracy: 100%
Train Epoch: 19 [193280/196000 (99%)] Relations accuracy: 80% | Non-relations accuracy: 100%
Train Epoch: 19 [194560/196000 (99%)] Relations accuracy: 89% | Non-relations accuracy: 100%
Train Epoch: 19 [195840/196000 (100%)] Relations accuracy: 91% | Non-relations accuracy: 100%

Test set: Relation accuracy: 90% | Non-relation accuracy: 99%
Train Epoch: 20 [193280/196000 (99%)] Relations accuracy: 91% | Non-relations accuracy: 100%
Train Epoch: 20 [194560/196000 (99%)] Relations accuracy: 97% | Non-relations accuracy: 100%
Train Epoch: 20 [195840/196000 (100%)] Relations accuracy: 95% | Non-relations accuracy: 100%

Test set: Relation accuracy: 89% | Non-relation accuracy: 99%

From the article in the "Dealing with pixels" case:

So, after convolving the image, each of the d^2 k-dimensional cells in the d × d feature maps was tagged with an arbitrary coordinate indicating its relative spatial position, and was treated as an object for the RN.

Also, the author (/u/asantoro) confirmed on reddit that objects were of the form:
[x, y, v_1, v_2, ..., v_k] where k is the number of kernels and the range of the coordinates x,y doesn't matter.
(Reddit link)

So I think in the model, object coordinates should be added to oi and oj.
https://github.com/kimhc6028/relational-networks/blob/master/model.py#L53

for i in range(25):
    oi = x[:,:,i/5,i%5]
    for j in range(25):
        oj = x[:,:,j/5,j%5]
        x_ = torch.cat((oi,oj,qst), 1)
        x_ = self.g_fc1(x_)

I believe this should improve performance on questions where the spatial relationship between objects is important (closest, furthest, ...).