Evaluatiuon Exception error: Your results will NOT be comparable with results in the paper or online. about synsin HOT 2 CLOSED

Sorry that Matterport is a pain -- we would have released the images but weren't allowed to If the check is failing it means the images being generated on the fly are not the ones we used, so the numbers you get when you comment out the line are not comparable (and this is what I would expect -- hence having the check).

You have two choices:

1. You could just use the new set of images (but then please run an evaluation on KITTI or the RealEstate dataset to ensure that your version of pytorch is running the models correctly)

2. Try to diagnose why you're getting different generated images. My hunch is that it's because you're not loading in the dataset splits correctly. These splits are https://github.com/facebookresearch/synsin/tree/master/data/scene_episodes and they decide the environment to load in. These splits are generated if they are not found here, so the code would still run but the splits would be wrong. I would check that the right splits are being found in that part of the code. I hope that helps!

from synsin.

I also encountered this error, and I figured out that this was a habitat version issue.

After switching back to commits

habitat-sim: d383c2011bf1baab2ce7b3cd40aea573ad2ddf71
habitat-api: e94e6f3953fcfba4c29ee30f65baa52d6cea716e

as mentioned in #2 (comment), now the evaluation runs as expected.

from synsin.