Skip to content

Add full LaMAR, Spires, Hilti and stereo benchmarks - #56

Merged
Zador-Pataki merged 2 commits into
mainfrom
zador/expanded-benchmark-datasets
Nov 4, 2026
Merged

Zador-Pataki merged 2 commits into
mainfrom
zador/expanded-benchmark-datasets

Conversation

@Zador-Pataki

@Zador-Pataki Zador-Pataki commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Add more datasets and camera setups to the standard benchmark commands.

Changes

  • LaMAR: use complete trajectories with available ground truth by default; keep sample-300 for the paper trajectories.
  • Oxford Spires: rectified monocular, native fisheye and three-camera rig modes across 13 scenes.
  • Hilti: 360° panoramas from 30 sequences.
  • EuRoC / ETH3D-SLAM: stereo preparation that reuses existing cam0 images.

Includes source inventories, preparation, benchmark definitions and usage instructions. Hilti stitching can introduce near-field parallax.

Benchmark results: additional datasets

LaMAR results cover 53 campaign trajectories, versus 128 recordings in the new preparation. Spires monocular results cover 10 of 13 scenes.

LaMAR complete · 53 scenes · W-AUC at 5%

Method ATE (m) ↓ 10 m 25 m 50 m 100 m Full
RoMaV2 1.452 92.42 92.83 92.38 92.59 89.76

Spires rectified monocular · 10 scenes · W-AUC at 5%

Method ATE (m) ↓ 10 m 25 m 50 m 100 m Full
RoMaV2 0.130 98.98 99.10 99.11 99.10 99.54

Hilti forward monocular · 30 scenes · W-AUC at 5%

Method ATE (m) ↓ 10 m 25 m 50 m 100 m Full
RoMaV2 0.822 93.96 93.96 93.29 92.44 94.46

Hilti three-view · 30 scenes · W-AUC at 5%

Method ATE (m) ↓ 10 m 25 m 50 m 100 m Full
RoMaV2 0.206 94.80 96.22 96.94 97.21 98.13

Spires three-camera rig · 13 scenes · W-AUC at 5%

Method ATE (m) ↓ 10 m 25 m 50 m 100 m Full
RoMaV2 0.119 99.15 99.24 99.26 99.28 99.59

@Zador-Pataki
Zador-Pataki marked this pull request as ready for review November 4, 2026 01:58
@Zador-Pataki
Zador-Pataki changed the base branch from zador/ate-benchmark-reporting to main November 4, 2026 21:04
@Zador-Pataki
Zador-Pataki merged commit 0f39f8d into main Nov 4, 2026
1 check passed
@Zador-Pataki
Zador-Pataki deleted the zador/expanded-benchmark-datasets branch November 4, 2026 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant