set this up https://www.appveyor.com

port a keras example:

https://keon.io/deep-q-learning/
or
https://github.com/matthiasplappert/keras-rl/tree/master/rl/agents

need to get a logo designed soon. we need to get some concept art designed for this.
probably an image with some combination of touch / senses and brains/ artificial intelligence.

need to figure out how to model this.

how many steps do we need to get to first touch and then turn around that object.

While the wand is in contact with the object the movement slows down, giving the appearance that the wand is stuck on the object. This is noticeable during the rendering process.

Wand should not show an appreciable slowdown while in contact with the object.

Related to the use of the computeProjectionMatrixFOV and getCameraImage functions. These introduce overhead that causes the time between successive calls to _step, thus giving the appearance of a slowdown.

right now it displays "bullet open gl test", we want to change it to sensenet

add a new expierence_replay version

add some basic tests:

can load environment
can create new environment
hand registers touch

https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5

this is a pure numpy example. port it to work on sensenet just like our reinforce example:
https://github.com/jtoy/sensenet/blob/master/agents/reinforce.py

hand_env
versioned system

need to debug pyramids , seems like we see/touch certain ones due to angle of the pyramid, no idea why.

all of our dependencies work on windows so this code should work on windows. We want to officially confirm it works. We need someone with a windows machine to run the "cd tests && pytest" to confirm it works on windows. Ideally we would have a test that gets run on every commit to make sure window is working. Unfortunately, travis, our open source testing facility doesn't support windows, can we use another automated test system to make sure windows support works?

can we make it touch for every episode?

i think the new RNN functionality kills it:
https://discuss.pytorch.org/t/training-rnn-on-gpu/3574/5

base_linkepisode: 48

1 touches in current episode <<
env.class_label: 9
Traceback (most recent call last):
File "reinforce.py", line 243, in
output = cnn_lstm(observed_touches) # Prediction
File "/home/jtoy/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 206, in call
result = self.forward(*input, **kwargs)
File "reinforce.py", line 146, in forward
rnn_out, rnn_hidden = self.rnn(inp.view(1, 1, -1), rnn_hidden)
File "/home/jtoy/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 206, in call
result = self.forward(*input, **kwargs)
File "/home/jtoy/anaconda3/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 91, in forward
output, hidden = func(input, self.all_weights, hx)
File "/home/jtoy/anaconda3/lib/python3.6/site-packages/torch/nn/_functions/rnn.py", line 343, in forward
return func(input, *fargs, **fkwargs)
File "/home/jtoy/anaconda3/lib/python3.6/site-packages/torch/autograd/function.py", line 202, in _do_forward
flat_output = super(NestedIOFunction, self)._do_forward(*flat_input)
File "/home/jtoy/anaconda3/lib/python3.6/site-packages/torch/autograd/function.py", line 224, in forward
result = self.forward_extended(*nested_tensors)
File "/home/jtoy/anaconda3/lib/python3.6/site-packages/torch/nn/_functions/rnn.py", line 285, in forward_extended
cudnn.rnn.forward(self, input, hx, weight, output, hy)
File "/home/jtoy/anaconda3/lib/python3.6/site-packages/torch/backends/cudnn/rnn.py", line 239, in forward
fn.hx_desc = cudnn.descriptor(hx)
File "/home/jtoy/anaconda3/lib/python3.6/site-packages/torch/backends/cudnn/init.py", line 304, in descriptor
descriptor.set(tensor)
File "/home/jtoy/anaconda3/lib/python3.6/site-packages/torch/backends/cudnn/init.py", line 110, in set
self, _typemap[tensor.type()], tensor.dim(),
KeyError: 'torch.FloatTensor'

our collisions use convex hull shapes. we want to model more accurately with concave shapes. we need to use VHACD on our objects. see this ticket for more information: bulletphysics/bullet3#1324

right now each action is defined as an integer and then I create a lookup table/case/if statements to process the action, if the order of actions changes, the code is broken, need a more robust system to deal with actions.

better error if path to dataset not found, this is current error:

python reinforce.py --data_path=../concave_objects --render --obj_type=obj
pybullet build time: Nov 30 2017 10:16:07
Vendor: NVIDIA Corporation
Renderer: NVIDIA GeForce GT 750M OpenGL Engine
Version: 4.1 NVIDIA-10.4.2 310.41.35f01
GLSL: 4.10
b3Printf: Selected demo: Physics Server
startThreads creating 1 threads.
starting thread 0
started thread 0
MotionThreadFunc thread started
Traceback (most recent call last):
File "reinforce.py", line 192, in
env = SenseEnv(vars(args))
File "../env.py", line 47, in init
self.load_object()
File "../env.py", line 78, in load_object
stlfile = files[random.randrange(0,files.len())]
File "/Users/jtoy/miniconda3/lib/python3.6/random.py", line 198, in randrange
raise ValueError("empty range for randrange() (%d,%d, %d)" % (istart, istop, width))
ValueError: empty range for randrange() (0,0, 0)
numActiveThreads = 0
stopping threads
stopThreads: Thread 0 used: 1
Thread with taskId 0 exiting
Thread TERMINATED
destroy semaphore
semaphore destroyed
destroy main semaphore
main semaphore destroyed

it works on certain angles, but not all angles. need to test and verify this. this is how the could should work:

You will need to compute the viewmatrix, based on the index finger world transform matrix. You can get this using the pybullet.getLinkState API. Then extract the position and orientation form that, and build the view matrix using pybullet.calculateViewMatrix. It is a bit complex, the racecar example (https://github.com/bulletphysics/bullet3/blob/449c8afc118a7f3629bc940c304743a084dcfac6/examples/pybullet/gym/pybullet_envs/bullet/racecarZEDGymEnv.py#L95) computes the camera based on the chassis. The finger world transform 3x3 matrix is basically fwd, left, up vectors.

we need to fully document:
environment API
agent API
benchmark API
dataset

we can't use resetBasePositionAndOrientation because that breaks the physics simulation.
don't use "resetBasePositionAndOrientation" while simulating, it should be only if you 'reset' the simulation at the start of the simulation.

Instead, create a 'fixed' constraint, and set its transform. See for example vrhand.py ( https://github.com/bulletphysics/bullet3/blob/master/examples/pybullet/examples/vrhand.py )

hand_cid = p.createConstraint(hand,-1,-1,-1,p.JOINT_FIXED,[0,0,0],[0.1,0,0],[0.500000,0.300006,0.700000],ho)

Then instead of using resetBasePositionAndOrientation, you use

p.changeConstraint(hand_cid,e[POSITION],e[ORIENTATION], maxForce=50)

(pick your max force as suitable)

In the TouchWandEnv it takes the wand approximately 30,000 moves to touch the object.

In the HandEnv it takes the hand only hundreds of moves to touch the object.

I suspect this is related to the interplay between the maxForce (in the changeConstraint), the mass of the agent, and the self.move parameter

These numbers of actions to touch the object should be standardized to enable the use of a universal max_steps parameter