ugo-nama-kun / dqn-chainer Goto Github PK

Hello, ugo-nama-kun,

Thank you very much for your great code of DQN in chainer.
It helps my research.
I would like to get suggestion to change CPU implementation.

According to my understanding, CuPy has compatibility for NumPy so that I just remove to_gpu() method from your code. However that code cause error as follows.
So far I am not sure this is limitation of CuPy or my misunderstanding.
Could you please suggest about this issue?

=== Error when it remove to_gpu() method from the code ===
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/rlglue/agent/ClientAgent.py", line 51, in onAgentStart
action = self.agent.agent_start(observation)
File "dqn_agent_nature_cpu.py", line 216, in agent_start
action, Q_now = self.DQN.e_greedy(state_, self.epsilon)
File "dqn_agent_nature_cpu.py", line 167, in e_greedy
Q = self.Q_func(s)
File "dqn_agent_nature_cpu.py", line 144, in Q_func
h1 = F.relu(self.model.l1(state / 254.0)) # scale inputs in [0.0 1.0]
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/chainer-1.3.0-py2.7.egg/chainer/function.py", line 174, in call
outputs = self.forward(in_data)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/chainer-1.3.0-py2.7.egg/chainer/function.py", line 238, in forward
return self.forward_cpu(inputs)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/chainer-1.3.0-py2.7.egg/chainer/functions/connection/convolution_2d.py", line 165, in forward_cpu
y = numpy.tensordot(self.col, self.W, ((1, 2, 3), (1, 2, 3)))
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/numeric.py", line 1284, in tensordot
a, b = asarray(a), asarray(b)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/numeric.py", line 474, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: object array method not producing an array

Thanks,
brgyk.

Hi.

I'm using

ALE 0.5.0
RL-Glue 3.04 Build 909
and very recent chainer

after this message (in experiment_ale.py)

DQN-ALE Experiment starting up!
RL-Glue Python Experiment Codec Version: 2.02 (Build 738)
Connecting to 127.0.0.1 on port 4096...
Freeze learning for Evaluation
Evaluation :: 945 steps -21.0 total reward
DQN is Learning
Episode 1 825 steps -21.0 total reward
DQN is Learning
Episode 2 1066 steps -21.0 total reward
DQN is Learning
Episode 3 1041 steps -20.0 total reward
DQN is Learning
Episode 4 1318 steps -19.0 total reward
DQN is Learning
Episode 5 885 steps -21.0 total reward
DQN is Learning
Episode 6 1414 steps -18.0 total reward
DQN is Learning
Episode 7 764 steps -21.0 total reward
DQN is Learning
Episode 8 1039 steps -20.0 total reward
DQN is Learning
Episode 9 885 steps -21.0 total reward
Freeze learning for Evaluation
Evaluation :: 884 steps -21.0 total reward

the agent part get error like below:

Traceback (most recent call last):
File "dqn_agent_nature.py", line 303, in
AgentLoader.loadAgent(dqn_agent())
File "/home/deruci/anaconda/lib/python2.7/site-packages/rlglue/agent/AgentLoad er.py", line 58, in loadAgent
client.runAgentEventLoop()
File "/home/deruci/anaconda/lib/python2.7/site-packages/rlglue/agent/ClientAge nt.py", line 144, in runAgentEventLoop
switchagentState
File "/home/deruci/anaconda/lib/python2.7/site-packages/rlglue/agent/ClientAge nt.py", line 139, in
Network.kAgentStep: lambda self: self.onAgentStep(),
File "/home/deruci/anaconda/lib/python2.7/site-packages/rlglue/agent/ClientAge nt.py", line 62, in onAgentStep
action = self.agent.agent_step(reward, observation)
File "dqn_agent_nature.py", line 246, in agent_step
self.DQN.experienceReplay(self.time)
File "dqn_agent_nature.py", line 133, in experienceReplay
loss, _ = self.forward(s_replay, a_replay, r_replay, s_dash_replay, episode_ end_replay)
File "dqn_agent_nature.py", line 89, in forward
loss = F.mean_squared_error(td_clip, zero_val)
File "/home/deruci/anaconda/lib/python2.7/site-packages/chainer/functions/mean _squared_error.py", line 61, in mean_squared_error
return MeanSquaredError()(x0, x1)
File "/home/deruci/anaconda/lib/python2.7/site-packages/chainer/function.py", line 164, in call
self._check_data_type_forward(in_data)
File "/home/deruci/anaconda/lib/python2.7/site-packages/chainer/function.py", line 191, in _check_data_type_forward
self.check_type_forward(in_type)
File "/home/deruci/anaconda/lib/python2.7/site-packages/chainer/functions/mean _squared_error.py", line 17, in check_type_forward
in_types[0].shape == in_types[1].shape
File "/home/deruci/anaconda/lib/python2.7/site-packages/chainer/utils/type_che ck.py", line 457, in expect
expr.expect()
File "/home/deruci/anaconda/lib/python2.7/site-packages/chainer/utils/type_che ck.py", line 428, in expect
'{0} {1} {2}'.format(left, self.inv, right))
chainer.utils.type_check.InvalidType: Expect: in_types[1].dtype == <type 'numpy. float32'>
Actual: float64 != <type 'numpy.float32'>

and in ALE, I can see

Segmentation fault (core dumped)

I used same command as you suggested.
Is this problem happened because of the difference of ALE/RL-Glue/Chainer version?

Or

Can you tell me how I can fix this?

Thank you.

Hello,
Thanks for sharing this great code of chainer based DQN.

I recently started to use chainer. The code works great for me and I would like to implement a critic-actor architecture based on your DQN code. I don't know whether it is strange to ask question like this here, since I couldn't find appropriate forum to ask about chainer. But it will be really appreciated if help can be offered.

I can see in the code that the target model is updated as follow which directly copy from its model:

self.model_target = copy.deepcopy(self.model)

If I want the target to be updated slower base on this paper:

θ' ← τθ + (1 −τ )θ'

Because the θ and θ' is the weights of the model and target_model, I was thinking of doing:

self.target_model.W.data = tau*self.model.W.data + (1-tau)*target_model.W.data

Is this the right way of doing it? or any better suggestions of doing it using chainer?
Many thanks