Through the study on OpenAI's model, here is some useful information for developers who wrote their own version of mlstm and try to import OpenAI's model paramters. In mlstm function in encoder.py, defines the tensors' name, this is the baseline.

1. Computation Graphic and tensor
   Under the name scope model, there are three sub name scope:

   • embedding
     • tensors: w
   • out
     • tensors: w, b
   • rnn
     • tensors: b, gh, gmb, gmx, gx, wh, wmh, wmx, wx
       The tensors are listed as follow:
   1. tensor_name: model/embedding/w
   2. tensor_name: model/out/b
   3. tensor_name: model/out/w
   4. tensor_name: model/rnn/b
   5. tensor_name: model/rnn/gh
   6. tensor_name: model/rnn/gmh
   7. tensor_name: model/rnn/gmx
   8. tensor_name: model/rnn/gx
   9. tensor_name: model/rnn/wh
   10. tensor_name: model/rnn/wmh
   11. tensor_name: model/rnn/wmx
   12. tensor_name: model/rnn/wx

2. Table for the correlation between tensor and .npy files
   For detailed information about each tensor and which .npy it is correlated, please check the table
   Line of code follows the openAI's orignal code repo.

Hopyfully this would help.

Hi Stéphane @guillitte ,

thanks for sharing this, its really very interesting 👍

I don't know where to get the file, mlstm_ns.pt?

On the front of the repo it says,

Click on release to get model file mlstm_ns.pt

Where is release?

Thanks for your help,

Ajay :)

I am trying to test the visualisation process and this gives me weird results
see figure :

The code in models.py constructs the graph in a very sleek way. Is it possible to see how you transformed the weights into mlstm_ns.pt too?

Hi,

I converted the model to run on cpu using,

checkpoint = torch.load(opt.load_model)
embed = checkpoint['embed']
rnn = checkpoint['rnn']

embed = embed.cpu()
rnn = rnn.cpu()

because I'm interested in running unsupervised trained networks on low resource hardware. When I sample the model with command,

python visualize.py -seq_length 1000 -load_model mlstm_ns.pt -temperature 0.4 -neuron 2388 -init "I couldn't figure out"

The sentiment is not consistent,

Have you got any ideas how I could try to fix it please? I'd like to get the model to run on the cpu somehow?

Thanks a lot 👍

When I sample the model with command:

python visualize.py -seq_length 1000 -cuda -load_model mlstm_ns.pt -temperature 0.4 -neuron 2388 -init "I couldn't figure out"

and the process returned an error:

does the code can't run on python-2.7 or other problem?

thanks a lot.

Hello, I have created this Docker image for the original sentiment neuron that helps to test prediction of the sentiment feature as well as the generative model. I would like to add training, so I was starting from your Pytorch version.
How is the train and validation dataset format? Did you use the Amazon Reviews Full or Polarity Dataset for training i.e. the kaggle version here?

Thank you very much.

Hi, could you change your readme or include the params as the appear in the openai paper.
Specifically could you set seq_length to 256 and batch_size to 128, or at least mention that those are the params openai used in the paper.
Would've saved me some head scratching as to why I couldn't reproduce their results.
TY

Hi,

I am trying to clone the function of generating heatmap , Can you please point out why my lines isn't breaking and why my font is so small.

https://github.com/yashkumaratri/testrepo/blob/master/heatmap.ipynb

I am trying to use the repository for training a new model but I got the following error

/Users/virtualenv-dir/py3/bin/python /Users/Documents/Codes1/pytorch-sentiment-neuron-master/convert_to_cpu.py /Users/virtualenv-dir/py3/lib/python3.6/site-packages/torch/serialization.py:325: SourceChangeWarning: source code of class 'torch.nn.modules.linear.Linear' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = Trueand use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) Traceback (most recent call last): File "/Users/Documents/Codes1/pytorch-sentiment-neuron-master/convert_to_cpu.py", line 18, in <module> checkpoint = torch.load(opt.load_model) File "/Users/virtualenv-dir/py3/lib/python3.6/site-packages/torch/serialization.py", line 267, in load return _load(f, map_location, pickle_module) File "/Users/virtualenv-dir/py3/lib/python3.6/site-packages/torch/serialization.py", line 426, in _load result = unpickler.load() File "/Users/virtualenv-dir/py3/lib/python3.6/site-packages/torch/serialization.py", line 395, in persistent_load data_type(size), location) File "/Users/virtualenv-dir/py3/lib/python3.6/site-packages/torch/serialization.py", line 87, in default_restore_location result = fn(storage, location) File "/Users/virtualenv-dir/py3/lib/python3.6/site-packages/torch/serialization.py", line 69, in _cuda_deserialize return obj.cuda(device) File "/Users/virtualenv-dir/py3/lib/python3.6/site-packages/torch/_utils.py", line 68, in _cuda with torch.cuda.device(device): File "/Users/virtualenv-dir/py3/lib/python3.6/site-packages/torch/cuda/__init__.py", line 218, in __enter__ self.prev_idx = torch._C._cuda_getDevice() AttributeError: module 'torch._C' has no attribute '_cuda_getDevice'

It seems that the code only run with Cuda, could you please give me a hint to continue?

Thanks,
Mohammad

Thanks for your great work. Can you provide an example for feature extraction that corresponds to:

from encoder import Model

model = Model()
text = ['demo!']
text_features = model.transform(text)

in the original code. Or code that produces the .npy model files from the generated .pt file?
Thank you!

Thanks for this work. In the original project I have worked on a Dockerfile to support training on CPU/GPU with the generation as well.
Would be interesting to have in your Pytorch version a different output like json format, instead of visualizing the neuron output only.