OpenICC: An Open IMU and Camera Calibrator

WORK IN PROGRESS

I developed this repository to experiment with the accurate calibration of action cameras (e.g. GoPro cameras) to use them for geometric vision tasks like Structure-from-Motion, Photogrammetry and SLAM. Modern action cameras are equipped with various sensors like IMUs (accelerometer, gyroscope and magnetometer) and GPS. However the calibration data (e.g. camera projection and IMU to camera transformations) is not available.

This is where the OpenImuCameraCalibrator comes in. With this toolbox you can:

• Calibrate the intrinsics of your a GoPro camera. Supported camera models are:
  • Fisheye [6]
  • Division Undistortion [5]
  • Field-of-View [3]
  • Double Sphere [2]
  • Extended Unified [4]
  • Pinhole
  • RadTan pinhole
• Extract the meta data integrated in the MP4 video file (called telemetry data)
• Calibrate the Camera to IMU rotation matrix and find the dataset dependent time offset
• Perform full continuous time batch optimization to find the full transformation matrix between IMU and camera
• [Experimental] Calibrate the rolling shutter line delay

Results

This section provides some results for my two GoPro cameras (6 and 9). You can use this to verify your own results or use them as initial values for your application. So far I have been setting them to FullHD with wide FoV and 30/60 fps. This is probably the most common setting that people use.

Camera Calibration

IMU to Camera Calibration

Installation instructions

ToDo

1. Clone and build OpenCV >= 4.5.0

2. Install ceres 2.0 http://ceres-solver.org/installation.html

3. Clone and build the TheiaSfM fork.

git clone https://github.com/urbste/TheiaSfM
cd TheiaSfM && mkdir -p build && cd build
cmake .. && make -j4
sudo make install

4. Install nodejs (needed to extract GoPro telemetry)

sudo apt install nodejs npm

5. Build this project

git clone https://github.com/urbste/OpenImuCameraCalibrator
mkdir -p build && cd build && cmake ..
make -j

Example: Visual-Inertial Calibration of a GoPro Camera

For this example I am using a GoPro 9. To calibrate the camera and the IMU to camera transformation we will use the following script: python/run_gopro_calibration.py

0. Get the test data from here: link. Then you can skip step 1. & 2.

1. Data acquisition

   1.1 Print out the target: resource/board.png to a paper and attach it to something rigid (e.g. a wall). Measure the size of a black square in meter (e.g. 0.021m).

   1.2 Now record 3 videos.

   1.2.1 The first is to calibrate the camera. Move SLOWLY around the board. We do not want motion blur or the rolling shutter to influence the result. Record for about 20-30s. --> e.g. GH0000.MP4

   1.2.2 Place the GoPro on the floor or on a table and press record. Leave it there for 10-20s without touching it or walking around it. This video will be used to estimate the current IMU bias. We assume it to be fixed during the calibration. --> e.g. GH0001.MP4

   1.2.3 Finally record the last video. Again record the board and make sure that you have good lighting conditions. If possible set the shutter time of your GoPro the minimum (e.g. 1/480). If the board is barely visible your lighting condition is not good enough. Having a very fast shutter assures crisp corners and less motion blur. Also it should improve RS line delay calibration. --> e.g. GH0002.MP4

   • Excite all 3 axis -> 3 translation and 3 rotation.
   • Move fast, but not too fast (motion blur).
   • Make sure that most of the board is visible

2. Create the following folder structure:

MyDataset
|-- cam
|     |-- GH0000.MP4
|   imu_bias
|     |-- GH0001.MP4
|   cam_imu
|     |-- GH0002.MP4

3. Run the calibration

python python/run_gopro_calibration.py --path_calib_dataset=/your/path/MyDataset --checker_size_m=0.021 --image_downsample_factor=2 --camera_model=DIVISION_UNDISTORTION

Also check out all the other parameters you can set!

4. The spline calibration in the end should converge smoothly after 8-15 iterations. If not, your recordings are probably not good enough to perform a decent calibration. Also have a look at the final spline fit to the IMU readings:

Acknowlegements

This library would not have been possible without these great OpenSource projects:

• TheiaSfM - Camera models and optimization
• Basalt-Headers - Spline implementation and optimization
• Lie Group Cumulative B-Splines - Lie Splines
• InertialScale - IMU to camera time offset and rotation matrix initialization
• OpenCV - Well OpenCV ;)
• Sophus - C++ Lie groups
• GoPro-Telemetry - Great JavaScript GoPro telemetry extractor
• Kontiki - Spline error weighting, VISFM
• IMU-TK - Static multi pose imu calibration
• Allan variance - Gyro noise characterization

Literature and Code

Libraries

• [1] Theia Multiview Geometry Library: Tutorial & Reference

Camera models:

• [2] The Double Sphere Camera Model, V. Usenko and N. Demmel and D. Cremers, In 2018 International Conference on 3D Vision (3DV)
• [3] FoV model: F. Devernay and O. Faugeras. Straight lines have to be straight. Machine vision and applications, 13(1):14–24, 2001.
• [4] Extended unified: B. Khomutenko, G. Garcia, and P. Martinet. An enhanced unified camera model. IEEE Robotics and Automation Let- ters, 1(1):137–144, Jan 2016.
• [5] Division undistortion: "Simultaneous linear estimation of multiple view distortion" by Andrew Fitzgibbon, CVPR 2001.
• [6] Fisheye: Kannala, Juho, and Sami S. Brandt. "A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses." IEEE transactions on pattern analysis and machine intelligence 28.8 (2006): 1335-1340.

IMU calibration:

• [11] D. Tedaldi, A. Pretto and E. Menegatti, "A Robust and Easy to Implement Method for IMU Calibration without External Equipments". In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA 2014), May 31 - June 7, 2014 Hong Kong, China, Page(s): 3042 - 3049

Misc:

• [7] Mustaniemi J., Kannala J., Särkkä S., Matas J., Heikkilä J. "Inertial-Based Scale Estimation for Structure from Motion on Mobile Devices", International Conference on Intelligent Robots and Systems (IROS), 2017
• [8] Efficient Derivative Computation for Cumulative B-Splines on Lie Groups, C. Sommer, V. Usenko, D. Schubert, N. Demmel, D. Cremers, In 2020 Conference on Computer Vision and Pattern Recognition (CVPR)
• [9] Larsson, Viktor, Zuzana Kukelova, and Yinqiang Zheng. "Making minimal solvers for absolute pose estimation compact and robust." Proceedings of the IEEE International Conference on Computer Vision. 2017.
• [10] Hannes Ovrén and Per-Erik Forssén Spline Error Weighting for Robust Visual-Inertial Fusion In Proceedings of the IEEE on Computer Vision and Pattern Recognition (CVPR) June 2018

How to compare to Kalibr

Work-in-progress!

1. Download CDE package from here

2. Open AprilTag target on screen or print it from here

3. Create a AprilTag yaml file and enter size of tag

4. Find out your GoPro IMU noise params using allan variance

   • Record a >2h GoPro video where the GoPro is actually standing still. Use the lowest resolution and FPS setting, as we do not need the video feed for this
   • This will create multiple video files, so we need to extract the telemetry from each and merge it to a large one.
   • Put all video files in a single folder
   • Use python/allan_variance.py for this
   • Finally run fit_allan_variance binary on the large telemetry file
   • This will give you a value for each axis x-y-z of gyroscope and accelerometer
   • Average these values

5. Finally create a imu.yaml

Example from my GoPro 9

#Accelerometers
accelerometer_noise_density: 1.5e-02   # White noise
accelerometer_random_walk:   4.5e-04   # Bias instability

#Gyroscopes
gyroscope_noise_density:     1.1e-03   # White noise
gyroscope_random_walk:       3.0e-05   # Bias instability

rostopic:                    /imu0      #the IMU ROS topic
update_rate:                 200.0      #Hz (for discretization of the values above)

6. Use python/extract_for_kalibr_bagcreator.py to extract telemetry and single images

7. Run kalibr_bagcreator

8. Run kalibr_calibrate_cameras

Use omni-radtan model.

9. Run kalibr_calibrate_imu_camera

with --time-calibration

Working on / ToDo / Contributions Welcome

v0.1

• Code cleanup, add license header
• use different initialization for linear mode
• Use more generic JSON meta data interface, so that others cameras can be calibrated that come with different telemetry formats
• Write example for the calibration of a Smartphone
• Rolling shutter calibration -> first resolve problems with templated spline functions
• Add readout time as a optimizable parameter -> not working well, probably residual weighting needs to be improved
• Allan variance -> imu_utils
• Support AprilTag board from Kalibr to record same datasets
• Calibrate axis misalignment and so on with MutliPoseOptimization from imu_tk
• Include imu axis and scale in spline error functions
• Beautify logs
• Cleanup

v0.2

• Calibrate also imu parameters like axis misalignment
• Model bias over time with a R3 spline
• Pose estimation with UPNP or MLPnP to support arbitraty camera types. Right now: undistortion and then using vanilla PnP --> will lead to problems for Ultra Wide Angle fisheye lenses (e.g. GoPro Max or potentially Max lens mod)
• Pose estimation with RSPnP
• use only bearings in spline reproj error -> local tangent reprojection error
• Accurate checkerboard detector (OpenCV has findCheckerboardSB. I integrated a first version but results were weird. Re-check...)
• Extend to multi-camera systems
• Docker?
• Add more camera models -> Scaramuzza omni model, ...
• Integrate updated version of spline optimizer

misc

• Put together a little paper on how this all works

Citation

If this tool helped you and you are using it in your work consider citing it as follows for now:

@misc{OpenICC,
  author = {Steffen Urban},
  title = {OpenICC: An Open IMU and Camera Calibrator},
  howpublished = "\url{https://github.com/urbste/OpenImuCameraCalibrator}",
}