How long to sample (100Hz) to be able to use Allan deviation for Kalibr? about kalibr HOT 30 CLOSED

For us, 10 seconds datasets worked well, so you should probably be fine with the memory of your phone. You will need to excite the rotational degrees of freedom in order to render the position of the accelerometers observable. The "delta angles traveled" are less important here than angular velocities. At the same time, you are constrained by the range of your sensors (saturations) and the rolling shutter effect and motion blur, so please do not overdo it.
Also, if you are using the checkerboard target, please be careful to not induce excessive roll angles with respect to the initial pose of the phone, since this will cause the detection to flip. This is stated somewhere in the wiki as well, although probably not prominently enough.

from kalibr.

This is what I got after about 2 hours (tried to keep the temperature as stable as possible). Something looks weird though: the values for all axis starts going downwards towards the end of the chart, while they should go up only.

from kalibr.

This Allan deviation plot looks reasonable. Note that a.) the precision of the Allan deviation estimate naturally decreases for higher integration times \tau and b.) sensors rarely show a noise behavior that matches the "white noise + random walk" model you refer to (which would indeed lead to a monotonic increase for \tau -> \infty), but rather show flickering or other noise characteristics. If you want to make sure your math is right, simulate N sample paths using parameters deduced from this plot and run them through your Allan deviation computation.

Please indicate the units (g or m/s^2) and which accelerometer you are using. 2h should be enough for the accel you are using to be in the ballpark with the parameters.

from kalibr.

Hey @nikolicj thanks a ton for your answer, it clarified things for me.

The units are g and the accelerometer is the one mounted on iPhone 6s (unknown model).

So I think white noise in this case would be:
0.0001 g = 0.000981 m/s^2, then if we count in other types of noise, the real value would be 0.001 m/s^2. Does that sound correct?

Thanks!

from kalibr.

Great. The 6s has two accels (see http://www.macrumors.com/2014/09/26/iphone-6-6-plus-two-accelerometers/). One of which, Bosch's BMA280 has an "output noise density" of 120 ug/sqrt(Hz) http://www.mouser.com/ds/2/783/BST-BMA280-DS000-11_published-786496.pdf, which matches your observations.

It would be interesting, too, to have a look at the PSD of the noise, to see if the chip is properly configured. Try pwelch in MATLAB.

from kalibr.

The article refers to iPhone 6, but the components didn't seem to change for 6s. From what I understand the external accelerator is the more capable one and used at high frequencies, the other one is used only for things like screen flips.

This is what I get for the PSD of the x axis:

This is y:

And this is z:

Regarding the bias, how would I proceed to do the line fitting with those irregular +0.5 curves?

Thanks!

from kalibr.

For the bias I extrapolated a value of about 0.00004 g, does that sound right?

from kalibr.

Yes. Although I would consider to start with a higher value to make sure the bias fluctuations of the z-axis are covered. How about 0.0001 g/s/sqrt(Hz)? Are you running in "extended mode"?

from kalibr.

The value for the "white noise density" sounds about right, too. Again, consider increasing it a bit due to the poorer performance of the z axis.

from kalibr.

Interesting PSD plots. I was just asking out of curiosity. You can plot it in a more easy to read log-log fashion as follows:

% accel
for i=1:3
    [X_psd, freq_omega] = pwelch(dataset.imu.a_m(i,:) - ...
        mean(dataset.imu.a_m(i,:)), [500], [], [], ...
        1/dataset.procparam.imu.Ts,'onesided');
    dataset.imu.proc.a.psd(i,:) = X_psd';
    freq_a = freq_omega';
end
figure();
loglog(freq_omega, 1/2*dataset.imu.proc.a.psd(:,:)','LineWidth', 2);
grid on;
set(gca,'YMinorGrid','off')
xlabel 'frequency [Hz]'
ylabel 'PSD [m^2/s^4/Hz]'
title(['PSD Estimate Accelerometer']);
legend('PSD a_x', 'PSD a_y', 'PSD a_z');
hold on;
ax = axis;
plot([ax(1), ax(2)]', ...
    (dataset.procparam.imu.noise_density_a)^2*ones(2,1),'k--','LineWidth', 2);
text(ax(1), (dataset.procparam.imu.noise_density_a)^2, ' datasheet', ...
    'VerticalAlignment','top');

from kalibr.

@nikolicj yeah if you count that 0.0001 g/s/sqrt(Hz) sounds good. 0.000981 m/s^3/sqrt(Hz) if I have to use it as input for Kalibr.

What do you mean by extended mode? Kalibr extended mode? If yes, I'm currently not using that.

Also, I tried your code, but couldn't make it work (procparam missing).

Thanks for your input!

from kalibr.

Now, I tried to pass these parameters + subpixel camera calibration to the cam + imu calibration algorithm, but it doesn't converge to a value that makes sense (huge errors in pixels).

I am wondering if it is because I take the timestamp from the accelerometer and assign them to the gyro samples too, to build a single CSV file, while iOS is actually giving me different times for raw gyro and raw accelerometer. Should I treat them as separate IMUs in Kalibr (don't know if it's possible)?

Also, there are other two issues with timing:

1. images don't come with a timestamp, so I assign one as soon as I copy from the system buffer (probably not very precise);
2. IMU timestamps are not absolute, but an offset from when the phone was started, so at the beginning I record the offset from UNIX time zero to the time the phone was switched on (again, not precise).

Plus, when rotating the phone fast, the effects of rolling shutter probably start show to up, even if keeping exposure at 20ms.

Seems like there are a lot of potential issues. 😞

These are some of the results:

[0.0]: J: 5.95847e+09
[1]: J: 1.1175e+09, dJ: 4.84097e+09, deltaX: 6.82632, LM - lambda:100 mu:2
[2]: J: 4.50358e+08, dJ: 6.67142e+08, deltaX: 3.7417, LM - lambda:33.3333 mu:2
Last step was a regression. Reverting
[3]: J: 2.01675e+09, dJ: -1.56639e+09, deltaX: 12.4257, LM - lambda:14.8449 mu:2
Last step was a regression. Reverting
[4]: J: 5.21268e+08, dJ: -7.09101e+07, deltaX: 4.84288, LM - lambda:59.3797 mu:4
Last step was a regression. Reverting
[5]: J: 5.84057e+08, dJ: -1.33699e+08, deltaX: 0.352785, LM - lambda:475.037 mu:8
Last step was a regression. Reverting
[6]: J: 6.61022e+08, dJ: -2.10664e+08, deltaX: 0.0286516, LM - lambda:7600.6 mu:16
Last step was a regression. Reverting
[7]: J: 6.95414e+08, dJ: -2.45055e+08, deltaX: 0.0041007, LM - lambda:243219 mu:32
Last step was a regression. Reverting
[8]: J: 7.03457e+08, dJ: -2.53099e+08, deltaX: 3.61675e-06, LM - lambda:1.5566e+07 mu:64

After Optimization (Results)
==================
Reprojection error squarred (cam0):  mean 965.147373819, median 197.734136306, std: 2397.56431517
Gyro error squarred (imu0):          mean 2333.01142947, median 609.881854607, std: 4177.46879056
Accelerometer error squarred (imu0): mean 33192.0456533, median 971.489052822, std: 90968.1903704

PS: I saw you just recently merged the extended mode 👍

from kalibr.

Update: I tried with the latest Kalibr version and I'm now getting an error at optimization time:

Exception in thread block: [aslam::Exception] /home/xxx/kalibr_workspace/src/Kalibr/aslam_nonparametric_estimation/aslam_splines/src/BSplineExpressions.cpp:447: toTransformationMatrixImplementation() assert(_bufferTmin <= _time.toScalar() < _bufferTmax) failed [1.47042e+09 <= 1.47042e+09 < 1.47042e+09]: Spline Coefficient Buffer Exceeded. Set larger buffer margins!
Exception in thread block: [aslam::Exception] /home/xxx/kalibr_workspace/src/Kalibr/aslam_nonparametric_estimation/aslam_splines/src/BSplineExpressions.cpp:447: toTransformationMatrixImplementation() assert(_bufferTmin <= _time.toScalar() < _bufferTmax) failed [1.47042e+09 <= 1.47042e+09 < 1.47042e+09]: Spline Coefficient Buffer Exceeded. Set larger buffer margins!
[ERROR] [1470508014.979992]: Optimization failed!

This is with and without the --perform-synchronization option. Will try with a new dataset.

EDIT: I removed the option --time-calibration I was using and added the new --perform-synchronization, it now gives me a result much closer to a good one:

Residuals
----------------------------
Reprojection error (cam0) [px]:     mean 4.77586965267, median 3.76882441868, std: 3.80121534479
Gyroscope error (imu0) [rad/s]:     mean 0.226602683475, median 0.183862291526, std: 0.160863450646
Accelerometer error (imu0) [m/s^2]: mean 0.260732864143, median 0.233035823768, std: 0.160459316524

Transformation T_cam0_imu0 (imu0 to cam0, T_ci): 
[[ 0.01286937 -0.99982697  0.01343152  0.03877562]
 [-0.99970746 -0.01314062 -0.02030571  0.07883787]
 [ 0.02047869 -0.01316627 -0.99970359 -1.10895743]
 [ 0.          0.          0.          1.        ]]

Will still try with a better dataset.

from kalibr.

You may want to try both --perform-synchronization and --time-calibration and extend the time offset padding with --timeoffset-padding to avoid the aforementioned failure. In general, problems with kalibr are often linked to incorrect timing. So if you are not absolutely sure how timestamps get assigned in your system, this is likely something you may want to familiarize yourself with.

from kalibr.

OK I investigated and I was able to retrieve the actual time a frame is buffered and I also know that IMU time is based on the same clock of the camera time, so they are natively synchronized. Also just found out (yikes) I was passing acceleration in g and not in m/s^2 to the ros bag. There is still some other problem lingering, as I still have a very high mean error after optimization.

from kalibr.

Haha, it would have been interesting to see whether that flaw would have been taken care of by estimating an appropriate scaling factor for the acceleration measurements (with the scale-misalignment model).
You would still need --time-calibration since there are likely some fixed temporal offsets that have not been considered in your timestamping. The --timeoffset-padding option provides bounds within which the estimated offset may vary during optimization.
Please also note that you may want to stamp mid-exposure times rather than the time when a frame was buffered. Since you seem to be using an iPhone which employs a rolling shutter camera, you may further want to find settings that minimize the rolling shutter effect. kalibr currently exclusively supports a global shutter model for cameras.

from kalibr.

Oh I didn't think about using that option to fix the acceleration!
I read about time calibration in the paper, makes sense now.
I will try to see if I can get that mid-exposure time, I think I saw other timestamps were available.
In my latest dataset I limited exposure to 5ms (recording @25fps) and used smooth, not too fast movements, so that's probably the best I can do about the rolling shutter effect (thanks, cheap cameras).

My last try output:

After Optimization (Results)
==================
Normalized Residuals
----------------------------
Reprojection error (cam0):     mean 3.78914575605, median 3.11263126233, std: 3.00379387964
Gyroscope error (imu0):        mean 14.4189141963, median 12.8743218708, std: 8.72518261647
Accelerometer error (imu0):    mean 24.740970891, median 19.8095918077, std: 18.3911759649

Residuals
----------------------------
Reprojection error (cam0) [px]:     mean 3.78914575605, median 3.11263126233, std: 3.00379387964
Gyroscope error (imu0) [rad/s]:     mean 0.115351313571, median 0.102994574966, std: 0.0698014609318
Accelerometer error (imu0) [m/s^2]: mean 0.24740970891, median 0.198095918077, std: 0.183911759649

It seems like the accelerometer always has the biggest error in my tests.
Also I read somewhere in this repo that there shouldn't be shocks in the sensor data, what does that mean and what is the reason?

from kalibr.

I played with the IMU intrinsics like I saw someone did in another Github issue and by increasing the values a lot I got to this result:

Optimizing...
Using the block_cholesky linear system solver
Using the levenberg_marquardt trust region policy
Using the block_cholesky linear system solver
Using the levenberg_marquardt trust region policy
Initializing
Optimization problem initialized with 20495 design variables and 245377 error terms
The Jacobian matrix is 511462 x 92211
[0.0]: J: 1.52419e+06
[1]: J: 180437, dJ: 1.34375e+06, deltaX: 0.025662, LM - lambda:10 mu:2
[2]: J: 176096, dJ: 4340.63, deltaX: 0.169595, LM - lambda:3.33333 mu:2
[3]: J: 172599, dJ: 3496.9, deltaX: 0.168728, LM - lambda:1.11111 mu:2
Last step was a regression. Reverting
[4]: J: 172710, dJ: -111.241, deltaX: 1.1077, LM - lambda:0.37037 mu:2
[5]: J: 172430, dJ: 168.784, deltaX: 0.0786476, LM - lambda:1.48148 mu:4
[6]: J: 172388, dJ: 42.3816, deltaX: 0.62806, LM - lambda:0.493827 mu:2
[7]: J: 172380, dJ: 7.88923, deltaX: 0.850208, LM - lambda:0.422069 mu:2
[8]: J: 172376, dJ: 4.31439, deltaX: 0.855957, LM - lambda:0.41966 mu:2
[9]: J: 172375, dJ: 0.918037, deltaX: 1.95398, LM - lambda:0.275672 mu:2
[10]: J: 172374, dJ: 0.948613, deltaX: 1.8985, LM - lambda:0.277702 mu:2
[11]: J: 172373, dJ: 0.718097, deltaX: 1.87113, LM - lambda:0.277703 mu:2
[12]: J: 172373, dJ: 0.625771, deltaX: 1.8418, LM - lambda:0.277907 mu:2
[13]: J: 172372, dJ: 0.576889, deltaX: 1.80905, LM - lambda:0.27841 mu:2
[14]: J: 172371, dJ: 0.561614, deltaX: 1.77471, LM - lambda:0.279099 mu:2
[15]: J: 172371, dJ: 0.521357, deltaX: 1.74464, LM - lambda:0.279503 mu:2
[16]: J: 172370, dJ: 0.503345, deltaX: 1.71296, LM - lambda:0.280093 mu:2
[17]: J: 172370, dJ: 0.484225, deltaX: 1.68302, LM - lambda:0.280593 mu:2
[18]: J: 172369, dJ: 0.455516, deltaX: 1.65434, LM - lambda:0.281041 mu:2
[19]: J: 172369, dJ: 0.454299, deltaX: 1.62366, LM - lambda:0.281713 mu:2
[20]: J: 172369, dJ: 0.422945, deltaX: 1.59729, LM - lambda:0.282065 mu:2
[21]: J: 172368, dJ: 0.422121, deltaX: 1.56807, LM - lambda:0.282718 mu:2
[22]: J: 172368, dJ: 0.342584, deltaX: 1.54268, LM - lambda:0.283081 mu:2
[23]: J: 172367, dJ: 0.417034, deltaX: 1.47741, LM - lambda:0.287341 mu:2
[24]: J: 172367, dJ: 0.391942, deltaX: 1.4562, LM - lambda:0.287496 mu:2
[25]: J: 172367, dJ: 0.357142, deltaX: 1.43582, LM - lambda:0.287608 mu:2
[26]: J: 172366, dJ: 0.326921, deltaX: 1.41351, LM - lambda:0.287948 mu:2
[27]: J: 172366, dJ: 0.318994, deltaX: 1.38435, LM - lambda:0.289055 mu:2
[28]: J: 172366, dJ: 0.360864, deltaX: 1.35343, LM - lambda:0.290433 mu:2
[29]: J: 172365, dJ: 0.234726, deltaX: 1.3357, LM - lambda:0.290456 mu:2
[30]: J: 172365, dJ: 0.360399, deltaX: 1.23166, LM - lambda:0.300642 mu:2

After Optimization (Results)
==================
Normalized Residuals
----------------------------
Reprojection error (cam0):     mean 0.645174917522, median 0.513043867999, std: 0.510187283316
Gyroscope error (imu0):        mean 0.751256954926, median 0.62197885077, std: 0.519613501661
Accelerometer error (imu0):    mean 0.880641238104, median 0.735521759436, std: 0.576059955255

Residuals
----------------------------
Reprojection error (cam0) [px]:     mean 0.645174917522, median 0.513043867999, std: 0.510187283316
Gyroscope error (imu0) [rad/s]:     mean 0.030050278197, median 0.0248791540308, std: 0.0207845400664
Accelerometer error (imu0) [m/s^2]: mean 1.05676948572, median 0.882626111323, std: 0.691271946305

Seems like LM is converging fast. But does increasing IMU intrinsics make sense? The only thing I see wrong in the calibration is the position of the IMU compared to the camera on the X axis (25cm away). I will try with a longer dataset.

from kalibr.

Both, reprojection error and accelerometer residuals are still large. We commonly get reprojection errors that are at least 6 times smaller and accelerometer residuals which are about 10 times smaller. Are you using the scale-misalignment model?
Often, having to inflate noise values from what has been determined by a principled approach hints to an underlying problem of system design or understanding. In my personal experience, the aspect most difficult to grasp is timing, especially if you are using a black-box system like a phone.
Please make sure to excite all rotational degrees of freedom sufficiently, since only rotational motions will render the position of the accelerometers observable.
At last, not accounting for the rolling shutter effect could potentially yield a bias in these estimates as well.

from kalibr.

Hey, sorry for the late response: I tried scale-misalignment but the result was the same. Thanks for pointing out ways of improving the dataset. I tried with another but it wasn't good as well. I am limited by the phone memory for how much I can record so I probably don't excite the DoFs enough, I will have to work on that.

One question I have is, do I need to excite the accelerometer DoFs for a good range (distance) or intensity (force)? I assume exciting the gyro involves rotating for a while by a big angle. Could you point me to resources that explain this process?

I'll keep you updated. Thanks a ton.

from kalibr.

Oh that's gold! I was doing exactly the opposite: long slow movements. I will try your approach next. Also, I am using a big Apriltag pattern.

from kalibr.

That improved the results without the necessity of increasing noise parameters and now I use linear interpolation to get gyro values at accelerometer times (to cope with the ros sensor message format), but it still is not good enough. It's hard to strike a balance between rolling shutter and sensor range.

from kalibr.

Yes, that is true. You could try to combine bits and pieces from PR #65 (please see this paper for details) with the IMU calibration for rolling shutter/IMU calibration. We might also be able to offer another solution in the future, but for now, you are on your own.

Another fix that you may want to try is a higher frame rate (you should be able to crank that up to 60 fps in the iPhone 6), which may result in an overall decreased exposure time and line delay. For a decreased computational load, you could still consider omitting some of the images.

from kalibr.

OK will look into the rolling shutter logic, see if I can make it work. Still trying different combination of speed of rotation and dataset length.

Yeah, the framerate is 25fps, but I manually locked exposure to 3ms, which resulted in much less motion blur, but you can still see wide effects of the rolling shutter.

from kalibr.

@rehderj if I understood it right, you suggested to use that algorithm to calibrate the camera line delay and then use the same PnP function to locate features while doing cam + IMU yes?

from kalibr.

Well, I would do the following: Leave all that untouched, hoping for the best, and plainly try to exchange the reprojection error formulation to Luc's:
Just add bits and pieces from Luc's code into Paul's code to make it work. Please let me know, how that goes.

from kalibr.

Sounds exciting, looking forward to it. Will let you know how it goes.

from kalibr.

Haha, i get that response a lot. People are usually overly excited about all things calibration ;)

from kalibr.

That's the first step of a long journey ;)

from kalibr.

Going to close this thread and open a more relevant one.

from kalibr.