Eric Brachmann@at @ducha_aiki I think so, too. It might have been easier 10 years ago, and for some problems maybe it still is. If the methods are still crude for a given problem, any ground truth might do to at least indicate the right direction. But for visual reloc, and other problems, our tools are so advanced by now that the question of the correct ground truth becomes more pressing.
Dmytro Mishkin 🇺🇦@eric_brachmann @at Didn’t LaMAR give at least temporary answer to visloc GT now?)
@ducha_aiki @at Yeah, maybe. It might have pushed back the GT uncertainty enough such that we do not have to worry for now. But I do not think anyone has given a principled answer yet.
Interestingly, LaMAR does not claim to be more accurate than 7Scenes. Of course, that same accuracy is much harder to achieve on this type of scenes and scale. But if you evaluate with a 5cm threshold on LaMAR, you might be in trouble again. Is 10cm OK? I do not know.
@ducha_aiki @at With the #MapFreeReloc benchmark (https://research.nianticlabs.com/mapfree-reloc-benchmark) we went the opposite direction: We argue that current methods are so bad, and our evaluation thresholds are so large that using SfM ground truth is probably fine.
If people start working on this successfully, we might need a new dataset in two years :)
@eric_brachmann @at mapfree, or better say see-through localization is great new task for sure :)