lvsn / deeptracking Goto Github PK

View Code? Open in Web Editor NEW

35.0 12.0 22.0 71 KB

Deep 6 DOF Tracking

License: Other

Python 84.63% Lua 14.39% Shell 0.07% Vim Script 0.03% Dockerfile 0.87%

deeptracking's Introduction

deeptracking

Torch + python implementation of Deep 6-DOF Tracking

The code to parse a ply file is not provided, this framework needs an implementation of plyparser.py

Useful links

Generating data

Generating synthetic data

Run:

python generate_synthetic_data.py config_file.json

dependencies

cv2
numpy (tested with 1.13.3)
tqdm
pyOpenGL
glfw
numpngw

configuration

see this example file

Generating real data

Run:

python generate_real_data.py config_file.json

dependencies

cv2
numpy (tested with 1.13.3)
tqdm
pyOpenGL
glfw
numpngw

configuration

see this example file

Train

Run:

python train.py config_file.json

dependencies

Hugh Perkins's pytorch
scipy, skimage, numpy (tested with 1.13.3)
tqdm
numpngw
slackclient (could be removed)

configuration

see this example file

Test

Sensor

Will run the tracker with a sensor (kinect 2)

python test_sensor.py config_file.json

Sequence

Will run the tracker on a sequence (folder) and save the error in a file

python test_sequence.py config_file.json

dependencies

cv2
numpy (tested with 1.13.3)
Hugh Perkins's pytorch
pyOpenGL
glfw
numpngw
pyfreenect2 (for Kinect2 sensor)

configuration

see this example file

deeptracking's People

Contributors

Stargazers

Watchers

deeptracking's Issues

Thread error : 'image_size'

Hi,

I am trying to rerun this project. After preparing for the dataset, I activate train.py file to traing a torch model. The program blocked in line 189/190 with the following output:

Thread error : 'image_size'
Process Process-13: Traceback (most recent call last): File "/home/sia/anaconda3/envs/py36/lib/python3.6/multiprocessing/process.py

", line 258, in _bootstrap self.run() File "/home/sia/anaconda3/envs/py36/lib/python3.6/multiprocessing/process.py

", line 93, in run self._target(*self._args, **self._kwargs) File "/home/sia/deeptracking/deeptracking/data/parallelminibatch.py

", line 65, in worker_ batch = self.load_minibatch(task) File "/home/sia/deeptracking/deeptracking/data/dataset.py

", line 243, in load_minibatch return image_buffer, prior_buffer, label_buffer UnboundLocalError: local variable 'image_buffer' referenced before assignment

It seems to be associated with the parallelminibatch module.
But I don't know how to resolve it.

issues about aruco.BoardConfiguration,aruco.BoardDetector

@MathGaron as per the instruction i run the test_sensor.py and i find the error:it says there is no module of aruco.BoardConfiguration,aruco.BoardDetector and aruco.CameraParameters，do you use some specific aruco right?
BTW, i am using ubuntu14.04,python3.4,aruco3.4,opencv3.4and python-aruco,so it might be the vision of aruco issue?can you share some suggestion?many thanks

add code for background dataset cleaning

PANIC: unprotected error in call to Lua API

Sorry to ask a number of times.
Executing test gives the following error.

PANIC: unprotected error in call to Lua API (module '/home/usr/deeptracking/deeptracking/tracker/rgbd_tracker' not found:
	no field package.preload['/home/usr/deeptracking/deeptracking/tracker/rgbd_tracker']
	no file '/home/usr/.luarocks/share/lua/5.1//home/usr/deeptracking/deeptracking/tracker/rgbd_tracker.lua'
	no file '/home/usr/.luarocks/share/lua/5.1//home/usr/deeptracking/deeptracking/tracker/rgbd_tracker/init.lua'
	no file '/home/usr/torch/install/share/lua/5.1//home/usr/deeptracking/deeptracking/tracker/rgbd_tracker.lua'
	no file '/home/usr/torch/install/share/lua/5.1//home/usr/deeptracking/deeptracking/tracker/rgbd_tracker/init.lua'
	no file './/home/usr/deeptracking/deeptracking/tracker/rgbd_tracker.lua'
	no file '/home/usr/torch/install/lib/lua/5.1//home/usr/deeptracking/deeptracking/tracker/rgbd_tracker.lua'
	no file '/home/usr/torch/install/lib/lua/5.1//home/usr/deeptracking/deeptracking/tracker/rgbd_tracker/init.lua'
	no file '/home/usr/torch/install/lib//home/usr/deeptracking/deeptracking/tracker/rgbd_tracker.so'
	no file '/home/usr/.luarocks/lib/lua/5.1//home/usr/deeptracking/deeptracking/tracker/rgbd_tracker.so'
	no file '/home/usr/torch/install/lib/lua/5.1//home/usr/deeptracking/deeptracking/tracker/rgbd_tracker.so'
	no file '/home/usr/torch/install/lib//home/usr/deeptracking/deeptracking/tracker/rgbd_tracker.so'
	no file './/home/usr/deeptracking/deeptracking/tracker/rgbd_tracker.so'
	no file '/home/usri/torch/install/lib/lua/5.1/loadall.so')

Do you know what's wrong?

How to use dataset

hi MathGaron
I have a rudimentary question.
Although I was able to do normal learning, I could not learn with occlusion and background. As a result, tracking was not successful.
Could you tell me how to use occlusion model and background model (SUN 3 D model)?
In other words, I do not know the data to be included in "occluder_path" and "background_path" used in "train.py".

_aruco.cpython-35m-x86_64-linux-gnu.so

I would like to do various experiments using this.
so, I definitely want to realize the implementation.

I get an error when I do the 'test_sensor.py'

Traceback (most recent call last):
  File "test_sensor.py", line 11, in <module>
    from deeptracking.detector.detector_aruco import ArucoDetector
  File "/home/toshi/deeptracking/deeptracking/detector/detector_aruco.py", line 5, in <module>
    import aruco
  File "/usr/local/lib/python3.5/dist-packages/aruco.py", line 28, in <module>
    _aruco = swig_import_helper()
  File "/usr/local/lib/python3.5/dist-packages/aruco.py", line 24, in swig_import_helper
    _mod = imp.load_module('_aruco', fp, pathname, description)
  File "/usr/lib/python3.5/imp.py", line 242, in load_module
    return load_dynamic(name, filename, file)
  File "/usr/lib/python3.5/imp.py", line 342, in load_dynamic
    return _load(spec)
ImportError: /usr/local/lib/python3.5/dist-packages/_aruco.cpython-35m-x86_64-linux-gnu.so: undefined symbol: _ZN5aruco14CvDrawingUtils10draw3dCubeERN2cv3MatERNS_6MarkerERKNS_16CameraParametersEb

python-aruco Installation succeeded.
I think it is okay to check the link.

$ ldd /usr/local/lib/python3.5/dist-packages/_aruco.cpython-35m-x86_64-linux-gnu.so
	linux-vdso.so.1 =>  (0x00007ffc0b3e5000)
	libopencv_core.so.3.1 => /usr/local/lib/libopencv_core.so.3.1 (0x00007fd6cd6c4000)
	libaruco.so.3.0 => /usr/local/lib/libaruco.so.3.0 (0x00007fd6cd382000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fd6cd000000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd6cccf7000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fd6ccae1000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd6cc717000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd6cc4fd000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd6cc2f9000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd6cc0dc000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd6cbed4000)
	libopencv_calib3d.so.3.1 => /usr/local/lib/libopencv_calib3d.so.3.1 (0x00007fd6cbb62000)
	libopencv_ml.so.3.1 => /usr/local/lib/libopencv_ml.so.3.1 (0x00007fd6cb8a1000)
	libopencv_highgui.so.3.1 => /usr/local/lib/libopencv_highgui.so.3.1 (0x00007fd6cb694000)
	libopencv_imgproc.so.3.1 => /usr/local/lib/libopencv_imgproc.so.3.1 (0x00007fd6c9c51000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fd6ce692000)
	libopencv_features2d.so.3.1 => /usr/local/lib/libopencv_features2d.so.3.1 (0x00007fd6c9987000)
	libopencv_flann.so.3.1 => /usr/local/lib/libopencv_flann.so.3.1 (0x00007fd6c972e000)
	libgtk-x11-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgtk-x11-2.0.so.0 (0x00007fd6c90e3000)
	libgdk-x11-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgdk-x11-2.0.so.0 (0x00007fd6c8e2e000)
	libcairo.so.2 => /usr/lib/x86_64-linux-gnu/libcairo.so.2 (0x00007fd6c8b1a000)
	libgdk_pixbuf-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so.0 (0x00007fd6c88f8000)
	libgobject-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007fd6c86a5000)
	libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007fd6c8394000)
	libopencv_imgcodecs.so.3.1 => /usr/local/lib/libopencv_imgcodecs.so.3.1 (0x00007fd6c7e32000)
	libgmodule-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x00007fd6c7c2e000)
	libpangocairo-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpangocairo-1.0.so.0 (0x00007fd6c7a21000)
	libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007fd6c76e7000)
	libXfixes.so.3 => /usr/lib/x86_64-linux-gnu/libXfixes.so.3 (0x00007fd6c74e1000)
	libatk-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libatk-1.0.so.0 (0x00007fd6c72bc000)
	libgio-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0 (0x00007fd6c6f34000)
	libpangoft2-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpangoft2-1.0.so.0 (0x00007fd6c6d1e000)
	libpango-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpango-1.0.so.0 (0x00007fd6c6ad2000)
	libfontconfig.so.1 => /usr/lib/x86_64-linux-gnu/libfontconfig.so.1 (0x00007fd6c688f000)
	libXrender.so.1 => /usr/lib/x86_64-linux-gnu/libXrender.so.1 (0x00007fd6c6685000)
	libXinerama.so.1 => /usr/lib/x86_64-linux-gnu/libXinerama.so.1 (0x00007fd6c6482000)
	libXi.so.6 => /usr/lib/x86_64-linux-gnu/libXi.so.6 (0x00007fd6c6272000)
	libXrandr.so.2 => /usr/lib/x86_64-linux-gnu/libXrandr.so.2 (0x00007fd6c6067000)
	libXcursor.so.1 => /usr/lib/x86_64-linux-gnu/libXcursor.so.1 (0x00007fd6c5e5d000)
	libXcomposite.so.1 => /usr/lib/x86_64-linux-gnu/libXcomposite.so.1 (0x00007fd6c5c5a000)
	libXdamage.so.1 => /usr/lib/x86_64-linux-gnu/libXdamage.so.1 (0x00007fd6c5a57000)
	libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007fd6c5845000)
	libpixman-1.so.0 => /usr/lib/x86_64-linux-gnu/libpixman-1.so.0 (0x00007fd6c559d000)
	libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007fd6c52f3000)
	libpng12.so.0 => /lib/x86_64-linux-gnu/libpng12.so.0 (0x00007fd6c50ce000)
	libxcb-shm.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-shm.so.0 (0x00007fd6c4eca000)
	libxcb-render.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-render.so.0 (0x00007fd6c4cc0000)
	libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007fd6c4a9e000)
	libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007fd6c4896000)
	libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fd6c4626000)
	libjpeg.so.8 => /usr/lib/x86_64-linux-gnu/libjpeg.so.8 (0x00007fd6c43cd000)
	libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007fd6c41ab000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fd6c3f90000)
	libharfbuzz.so.0 => /usr/lib/x86_64-linux-gnu/libharfbuzz.so.0 (0x00007fd6c3d32000)
	libthai.so.0 => /usr/lib/x86_64-linux-gnu/libthai.so.0 (0x00007fd6c3b29000)
	libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fd6c3900000)
	libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007fd6c36fc000)
	libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007fd6c34f6000)
	libgraphite2.so.3 => /usr/lib/x86_64-linux-gnu/libgraphite2.so.3 (0x00007fd6c32d0000)
	libdatrie.so.1 => /usr/lib/x86_64-linux-gnu/libdatrie.so.1 (0x00007fd6c30c8000)

What do you think is the problem?

question about quaternion

Hi,

I have a question regarding the calculation of the quaternion.

In the function get_sample, you transform the initial_pose for a quaternion through the function to_paremeters.

def get_sample(self, index, image_buffer, prior_buffer, label_buffer, buffer_index):
    rgbA, depthA, initial_pose = self.load_image(index)

    prior_buffer[buffer_index] = initial_pose.to_parameters(isQuaternion=True)

The function to_parameters in turn calls the function euler2quat and this passes as parameters the angles a, b, c corresponding to the vector of the pose $[x, y, z, a, b, c]$. For the angle $a$ on the $x$ axis, $b$ the angle on the $y$ axis, and $c$ the angle on the $z$ axis, all in radians.

def to_parameters(self, isDegree=False, isQuaternion=False):
	qx, qy, qz, qw = ea.euler2quat(a, b, c)

However, the euler2quat function in angles.py receives the parameters:

def euler2quat(z=0, y=0, x=0)

z : scalar
- Rotation angle in radians around z-axis (performed first)
y : scalar
- Rotation angle in radians around y-axis
x : scalar
- Rotation angle in radians around x-axis (performed last)

and return:

quat : array shape (4,)
- Quaternion in w, x, y z (real, then vector) format

what is not equivalent to the parameters in the to_parameters function:

qx, qy, qz, qw = ea.euler2quat(a, b, c)

I would like to know if I understood the procedure well and if that is the expected result.

Thank you,

question about traning data

@MathGaron @jflalonde ,hi ,according to the paper of the awesome project and the code of the project. I still fell some confused of the training data generation.As far as I can know is that we use the 3D model texture file (the ply files)and the things in the generate_synthetic_example.json to get the dataset of input Xpred. And the generate_real_example.json material (include the raw_training images???)are used to get the dataset of input Xobs.But the paper just said we just need the 3D model file and the P pred,Pobs to generate the training dataset.BTW ,the data is generated as a pair like 0.pny 0n0.pny,so the first time i just think they are Xpred and Xobs respectively.But i just find the generate_synthetic_data.py do not use the Euler angle notation and do not have the word of previous pose mentioned.so I guess that the two generators just make part of the training data sepetively.am i right?and there is something unnormal is that this is some data generated : they are just away from the paper .

python2.7.6 run capture_data.py errors.

HI,I am trying to make the awesome work running in my computer. But when I just installed all the dependences I just find that I get this error ,I think it might be the reason I am using python2 ,any suggestions? Many appreciate your time ,thanks : -).
gaofei@gaofei-HP-Z420-Workstation:~/deeptracking$ python capture_data.py
Traceback (most recent call last):
File "capture_data.py", line 1, in
from deeptracking.detector.detector_aruco import ArucoDetector
ImportError: No module named detector.detector_aruco

question about training

Hi,

I have some questions about network training:

In the repository, there are 5 files 'tracker.lua' with some differences between them (rgb_tracker_squeeze, rgbd_quaternion_tracker, rgbd_tracker, rgbd_tracker_no_maxpool, rgbd_tracker_squeeze). I would like to know which of these versions was the one that you used for the training of the network.
In the paper, Deep 6-DOF Tracking, the design of the network is presented in Figure 3, however it is not evident how the quaternion is included in the model. I would like to know how the quaternion is used as input in training.
Finally, in the paper, Deep 6-DOF Tracking in section 5.2, a set of 180 captures is mentioned for training, to carry out a fine-tuning before real-life testing. What are those images? How can I get those images?

I would like to understand these issues in order to achieve the same results presented in your paper.

Thank you.

asking for the dataset.

hi, i just read the paper for the project .we need the dataset for the json file right ?
"data_augmentation":{
"rgb_noise": "4", # std of rgb gaussian noise
"depth_noise": "20", # std of depth gaussian noise
"occluder_path": "/path/to/occluder/dataset",
"background_path": "/path/to/background/dataset",
"blur_noise": "7", # max gaussian blur kernel size
"h_noise": "0.07", # max hue noise
"s_noise": "0.0", # max saturation noise
"v_noise": "0.2" # max intensity noise
},
should i dwonload all the SUN3D database? and can i replace the occluder with any other 3d models?
wish for your reply. have a nice day.

Hello,here

so,how do you think the result if without depth information?

still issues about generate data .

hi, @MathGaron as you mentioned that I need to have a look of the Opengl and I do reinstall everything in a new computer but i still get the same issues .here in pic1 I used to test the opengl it is fine:

and then i run it with the skull ply files then i get this : with no error no warning and what I think it can not be the json file issue as you can see in the picture .could it be the issue of plyparser or any thing you do not mention in the project?BTW,could you share the workflow of install dependences or just share the generated data .That can be of great help with my issue.

and in the plyparser.py this part never used in the code why should we import PlyElement?
wish for your feedback,I only have just 3 days for the project .
btw,can the generate black picture be the training data as it cotains the pose information?
"""
@staticmethod
def save_points(points, path):
vertex = np.zeros(points.shape[0], dtype=([('x', 'f4'), ('y', 'f4'), ('z', 'f4')]))
vertex.fill(255)
vertex['x'] = points[:, 0]
vertex['y'] = points[:, 1]
vertex['z'] = points[:, 2]
el = PlyElement.describe(vertex, 'vertex')
PlyData([el], text=ascii).write(path)
"""

Training Failed, Help Wanted!

Thanks for the awesome work!

I didn't generate data but used the data loader to download the original data. However, the traninig was failed.

I print the data['metaData'] in load() function at deeptracking/data/dataset.py, showing skull's raw_training data's info:

{'save_type': 'png', 'PairsQty': 0, 'frameQty': '173'}

Then I get error raised in deeptracking/data/dataset.py:

def load_minibatch(self, task):

and then obviously get compute_mean as all 0s. I think the error occurs because of get_sample() part that attempts to get pairs but get nothing. But I am not sure why this happens.

In the end, though I think there should be no connection, I get another error for tracker class:

[INFO] Setup Model
config: ,  <class 'PyTorchLua.RGBDTracker'> 0
LuaWrapper.__init__ RGBDTracker fromLua False args ('cuda', 'adam', 0)
Traceback (most recent call last):
  File "train.py", line 227, in <module>
    tracker_model = config_model(data, train_dataset)
  File "train.py", line 132, in config_model
    tracker_model = model_class('cuda', 'adam', gpu_device)
  File "/home/name/.local/lib/python3.5/site-packages/PyTorch-4.1.1_SNAPSHOT-py3.5-linux-x86_64.egg/PyTorchHelpers.py", line 20, in __init__
    PyTorchAug.LuaClass.__init__(self, splitName, *args)
  File "/home/name/.local/lib/python3.5/site-packages/PyTorch-4.1.1_SNAPSHOT-py3.5-linux-x86_64.egg/PyTorchAug.py", line 255, in __init__
    raise Exception(errorMessage)
Exception: attempt to call a nil value

The train.json is as follows: (class.lua was copied to the current path)

{
  "data_augmentation":{
      "rgb_noise": "4",
      "depth_noise": "20",
      "occluder_path": "aug_util/occluder",
      "background_path": "aug_util/background",
      "blur_noise": "7",
      "h_noise": "0.07",
      "s_noise": "0.0",
      "v_noise": "0.2",
      "channel_hide": "True"
    },

  "training_param":{
      "file": "class.lua",
      "learning_rate": "0.005",
      "learning_rate_decay": "1e-5",
      "weight_decay": "0",
      "input_size": "150",
      "linear_size": "50",
      "convo1_size": "24",
      "convo2_size": "48"
    },

  "logging":{
      "path": "log/output",
      "level": "DEBUG"
  },

  "session_name": "test001",
  "train_path": "/data/name/deeptrack/raw_training/skull/train",
  "valid_path": "/data/name/deeptrack/raw_training/skull/valid",
  "output_path": "checkpoint",
  "model_finetune": "",
  "minibatch_size": "128",
  "max_epoch": "30",
  "early_stop_wait_limit" : "5",
  "gpu_device" : "0",
  "image_size": "150"
}

Are there any suggestions? I followed the README.md to installed Torch and python wrapper for it. Thanks a lot! @MathGaron

lvsn / deeptracking Goto Github PK

deeptracking's Introduction

deeptracking

Useful links

Generating data

Generating synthetic data

dependencies

configuration

Generating real data

dependencies

configuration

Train

dependencies

configuration

Test

Sensor

Sequence

dependencies

configuration

deeptracking's People

Contributors

Stargazers

Watchers

Forkers

deeptracking's Issues

Recommend Projects

Recommend Topics

Recommend Org