This is hloc, a modular toolbox for state-of-the-art 6-DoF visual localization. It implements Hierarchical Localization, leveraging image retrieval and feature matching, and is fast, accurate, and scalable. This codebase won the indoor/outdoor localization challenges at CVPR 2020 and ECCV 2020, in combination with SuperGlue, our graph neural network for feature matching.

With hloc, you can:

• Reproduce our CVPR 2020 winning results on outdoor (Aachen) and indoor (InLoc) datasets
• Run Structure-from-Motion with SuperPoint+SuperGlue to localize with your own datasets
• Evaluate your own local features or image retrieval for visual localization
• Implement new localization pipelines and debug them easily 🔥

Hierachical Localization uses both image retrieval and feature matching

hloc requires Python >=3.6, PyTorch >=1.1, and COLMAP. Other minor dependencies are listed in requirements.txt. For pose estimation, we use pycolmap, which can be installed as:

pip install git+https://github.com/mihaidusmanu/pycolmap

This codebase includes external local features as git submodules – don't forget to pull submodules with git submodule update --init --recursive. Your local features are based on TensorFlow? No problem! See below for the steps.

We also provide a Docker image that includes COLMAP and other dependencies:

docker build -t hloc:latest .
docker run -it --rm -p 8888:8888 hloc:latest  # for GPU support, add `--runtime=nvidia`
jupyter notebook --ip 0.0.0.0 --port 8888 --no-browser --allow-root

You can install hloc as a package and import it in other projects as an external library:

python -m pip install -e .

The toolbox is composed of scripts, which roughly perform the following steps:

1. Extract SuperPoint local features for all database and query images
2. Build a reference 3D SfM model
   1. Find covisible database images, with retrieval or a prior SfM model
   2. Match these database pairs with SuperGlue
   3. Triangulate a new SfM model with COLMAP
3. Find database images relevant to each query, using retrieval
4. Match the query images with SuperGlue
5. Run the localization
6. Visualize and debug

The localization can then be evaluated on visuallocalization.net for the supported datasets. When 3D Lidar scans are available, such as for the indoor dataset InLoc, step 2. can be skipped.

Strcture of the toolbox:

• hloc/*.py : top-level scripts
• hloc/extractors/ : interfaces for feature extractors
• hloc/matchers/ : interfaces for feature matchers
• hloc/pipelines/ : entire pipelines for multiple datasets

We provide step-by-step guides to localize with Aachen, InLoc, and to generate reference poses for your own data using SfM. Just download the datasets and you're reading to go!

Have a look at pipeline_Aachen.ipynb for a step-by-step guide on localizing with Aachen. Play with the visualization, try new local features or matcher, and have fun! Don't like notebooks? You can also run all scripts from the command line.

The notebook pipeline_InLoc.ipynb shows the steps for localizing with InLoc. It's much simpler since a 3D SfM model is not needed.

We show in pipeline_SfM.ipynb how to run 3D reconstruction for an unordered set of images. This generates reference poses, and a nice sparse 3D model suitable for localization with the same pipeline as Aachen.

• Supported local feature extractors: SuperPoint, D2-Net, SIFT, and R2D2.
• Supported feature matchers: SuperGlue and the Nearest Neighbor matcher with ratio test and/or mutual check.
• Supported image retrieval: NetVLAD and AP-GeM/DIR.

Using NetVLAD for retrieval, we obtain the following best results:

Check out visuallocalization.net/benchmark for more details and additional baselines.

We provide in hloc/pipelines/ scripts to run the reconstruction and the localization on the following datasets: Aachen Day-Night (v1.0 and v1.1), InLoc, Extended CMU Seasons, RobotCar Seasons, 4Seasons, Cambridge Landmarks, and 7-Scenes.

If you report any of the above results in a publication, or use any of the tools provided here, please consider citing both Hierarchical Localization and SuperGlue papers:

@inproceedings{sarlin2019coarse,
  title     = {From Coarse to Fine: Robust Hierarchical Localization at Large Scale},
  author    = {Paul-Edouard Sarlin and
               Cesar Cadena and
               Roland Siegwart and
               Marcin Dymczyk},
  booktitle = {CVPR},
  year      = {2019}
}

@inproceedings{sarlin2020superglue,
  title     = {{SuperGlue}: Learning Feature Matching with Graph Neural Networks},
  author    = {Paul-Edouard Sarlin and
               Daniel DeTone and
               Tomasz Malisiewicz and
               Andrew Rabinovich},
  booktitle = {CVPR},
  year      = {2020},
}

External contributions are very much welcome. This is a non-exaustive list of features that might be valuable additions:

• handle unknown query intrinsics (extraction from EXIF + refinement in PnP)
• support for GPS (extraction from EXIF + guided retrieval)
• covisibility clustering for InLoc
• visualization of the raw predictions (features and matches)
• other local features or image retrieval

Created and maintained by Paul-Edouard Sarlin with the help of many.