Add full LaMAR, Spires, Hilti and stereo benchmarks

README.md

@@ -353,10 +353,12 @@ By default, prepared data, caches, and experiments live under
 where the command is run. To use other paths, create `./vidmap-paths.toml` or
 set `VIDMAP_<DATASET>_{DATA,CACHE,EXP,TESTSETS}_DIR`.
 
+Spires and Hilti require the ROS bag reader (`pip install -e '.[dataset-preparation]'`).
+
 Then select and prepare a benchmark dataset:
 
 ```bash
-DATASET=lamar  # or: euroc, eth3d_slam, crocodl
+DATASET=lamar  # or: euroc, eth3d_slam, crocodl, spires, hilti
 python -m vidmap.datasets.prepare --dataset "$DATASET"
 ```
 
@@ -366,7 +368,9 @@ prepares only the images referenced by its benchmark testset: sessions and
 images absent from that testset are not retained. Downloads resume after
 interruption and completed sequences are reused. Validated source archives are
 retained by default; pass `--delete-downloads` to remove them after successful
-preparation.
+preparation. EuRoC and ETH3D-SLAM prepare only cam0 by default; `--mode stereo`
+or `--mode all` prepares both cameras, reusing existing cam0 images in the same
+folder. For Spires and Hilti, add `--workers N` to prepare N recordings concurrently.
 
 <details>
 <summary>[Expected prepared dataset layouts - click to expand]</summary>
@@ -375,16 +379,24 @@ Storage for the supported evaluation inventories:
 
 | Dataset | Download | Prepared |
 | --- | ---: | ---: |
-| LaMAR | 45.2 GB | 24.0 GB |
-| EuRoC | 24.7 GB | 4.7 GB |
-| ETH3D-SLAM | 18.6 GB | 18.6 GB |
+| LaMAR (sample-300) | 45.2 GB | 24.0 GB |
+| LaMAR (complete) | 144.5 GB | 76.7 GB |
+| EuRoC (monocular) | 24.7 GB | 4.7 GB |
+| EuRoC (rig) | 24.7 GB | 9.6 GB |
+| ETH3D-SLAM (monocular) | 18.6 GB | 18.6 GB |
+| ETH3D-SLAM (rig) | 37.3 GB | 37.3 GB |
 | CroCoDL | 15.0 GB | 14.0 GB |
+| Oxford Spires (rectified monocular, default) | 89.7 GB | 29.3 GB |
+| Oxford Spires (fisheye monocular) | 89.7 GB | 15.8 GB |
+| Oxford Spires (rectified rig) | 89.7 GB | 89.8 GB |
+| Hilti 360° | 132.1 GB | 120 GB |
 
 Download sizes are the checksum-pinned source archives; prepared sizes cover
 the canonical dataset after conversion. LaMAR archives are session-level, so
 the download includes unused frames inside each required session ZIP; only the
-94,489 testset images are retained. Retaining downloads requires roughly the sum
-of both columns, plus conversion staging.
+selected testset images are retained. Retaining downloads requires roughly the sum
+of both columns, plus conversion staging. Hilti's prepared size is estimated from
+120 panoramas sampled across all 30 recordings.
 
 After preparation, the selected dataset is available under
 `./local/datasets/<dataset>/`, with its benchmark definitions under
@@ -404,6 +416,8 @@ local/
 | --- | --- |
 | LaMAR, ETH3D-SLAM, CroCoDL | `native` |
 | EuRoC | `rectified` |
+| Oxford Spires | `native`, `rectified` |
+| Hilti 360° | `panorama` |
 
 <details>
 <summary>[Examples for each dataset - click to expand]</summary>
@@ -418,24 +432,45 @@ local/datasets/
 ├── euroc/machine_hall/MH_01_easy/
 │   ├── preparation.json
 │   └── rectified/
-│       ├── images/cam0/...
+│       ├── images/cam0/...         # cam1/ when stereo is prepared
 │       └── rec/
 ├── eth3d_slam/plant_1/0/
 │   ├── preparation.json
 │   └── native/
+│       ├── images/cam0/...         # cam1/ when stereo is prepared
+│       └── rec/
+├── crocodl/ios-ARCHE_D2/ios_2023-07-12_20.46.14_000/
+│   ├── preparation.json
+│   └── native/
 │       ├── images/cam0/...
 │       └── rec/
-└── crocodl/ios-ARCHE_D2/ios_2023-07-12_20.46.14_000/
+├── spires/2024-03-12-keble-college-02/0/
+│   ├── preparation.json
+│   ├── native/
+│   │   ├── images/cam0/...
+│   │   └── rec/
+│   └── rectified/
+│       ├── images/
+│       │   ├── cam0/...
+│       │   ├── cam1/...
+│       │   └── cam2/...
+│       └── rec/
+└── hilti/floor_1_2025-05-05_run_1/0/
     ├── preparation.json
-    └── native/
+    └── panorama/
         ├── images/cam0/...
         └── rec/
 
 local/testsets/
 ├── lamar/CAB/sample.yaml
-├── euroc/machine_hall/all.yaml
-├── eth3d_slam/plant_1/all.yaml
-└── crocodl/ios-ARCHE_D2/complete.yaml  # Also sample-*.yaml
+├── euroc/machine_hall/mono.yaml
+├── eth3d_slam/plant_1/mono.yaml
+├── crocodl/ios-ARCHE_D2/complete.yaml  # Also sample-*.yaml
+├── spires/2024-03-12-keble-college-02/
+│   ├── undistorted-mono.yaml
+│   ├── fisheye-mono.yaml
+│   └── undistorted-rig.yaml
+└── hilti/floor_1_2025-05-05_run_1/panorama.yaml
 ```
 
 </details>
@@ -448,8 +483,7 @@ Select a benchmark with `--mode`; its testset specifies the recording, image var
 ### Run and evaluate
 
 ```bash
-# Use uncalib/base for datasets other than LaMAR.
-FRONTEND_CONF=uncalib/gtkf-base
+FRONTEND_CONF=uncalib/base
 MAPPING_CONF=uncalib/base
 # Omit --name below to derive the output name from the config pair.
 OUTPUT_NAME=default
@@ -471,9 +505,7 @@ skip only this final report; per-trajectory evaluation still runs.
 <details>
 <summary>[LaMAR - click to expand]</summary>
 
-Evaluate the default `sample` benchmark, a representative subset of each
-ground-truth trajectory as outlined in the paper. LaMAR selects `wate_auc` by
-default.
+LaMAR defaults to `complete` with `wate_auc` evaluation.
 
 ```bash
 python -m vidmap.run_for_benchmark \
@@ -482,12 +514,25 @@ python -m vidmap.run_for_benchmark \
   --mapping-conf uncalib/base
 ```
 
+For the trajectories used in the paper, prepare `sample-300` and run `sample`:
+
+```bash
+python -m vidmap.datasets.prepare -d lamar --profile sample-300
+
+python -m vidmap.run_for_benchmark \
+  --dataset lamar --mode sample \
+  --frontend-conf uncalib/gtkf-base \
+  --mapping-conf uncalib/base
+```
+
+Use separate dataset/testset roots to keep both profiles.
+
 </details>
 
 <details>
 <summary>[EuRoC - click to expand]</summary>
 
-Evaluate the complete EuRoC benchmark. EuRoC selects `pose_auc` by default.
+EuRoC defaults to `mono` with `pose_auc` evaluation.
 
 ```bash
 python -m vidmap.run_for_benchmark \
@@ -496,13 +541,21 @@ python -m vidmap.run_for_benchmark \
   --mapping-conf uncalib/base
 ```
 
+For calibrated stereo:
+
+```bash
+python -m vidmap.run_for_benchmark \
+  --dataset euroc --mode stereo \
+  --frontend-conf calib/stereo \
+  --mapping-conf calib/base
+```
+
 </details>
 
 <details>
 <summary>[ETH3D-SLAM - click to expand]</summary>
 
-Evaluate the complete ETH3D-SLAM benchmark. ETH3D-SLAM selects `pose_auc` by
-default.
+ETH3D-SLAM defaults to `mono` with `pose_auc` evaluation.
 
 ```bash
 python -m vidmap.run_for_benchmark \
@@ -511,6 +564,15 @@ python -m vidmap.run_for_benchmark \
   --mapping-conf uncalib/base
 ```
 
+For calibrated stereo:
+
+```bash
+python -m vidmap.run_for_benchmark \
+  --dataset eth3d_slam --mode stereo \
+  --frontend-conf calib/stereo \
+  --mapping-conf calib/base
+```
+
 </details>
 
 <details>
@@ -527,6 +589,47 @@ python -m vidmap.run_for_benchmark \
 
 </details>
 
+<details>
+<summary>[Oxford Spires - click to expand]</summary>
+
+Oxford Spires has 13 scenes and defaults to rectified monocular images
+(`undistorted-mono`). Select a mode with `--mode`.
+
+| Mode | Frontend config | Mapping config |
+| --- | --- | --- |
+| `undistorted-mono` (default) | `uncalib/base` | `uncalib/base` |
+| `undistorted-mono` | `calib/base` | `calib/base` |
+| `fisheye-mono` | `uncalib/fisheye` | `uncalib/geocalib` |
+| `undistorted-rig` | `calib/rig` | `calib/base` |
+
+```bash
+python -m vidmap.datasets.prepare --dataset spires --mode undistorted-rig
+
+python -m vidmap.run_for_benchmark \
+  --dataset spires --mode undistorted-rig \
+  --frontend-conf calib/rig --mapping-conf calib/base
+```
+
+Evaluation uses scale-aligned WATE-AUC on cam0.
+
+</details>
+
+<details>
+<summary>[Hilti 360° - click to expand]</summary>
+
+Hilti has 30 sequences and defaults to `panorama` mode. Select perspective views
+with [`rig.projection`](docs/cameras.md#panoramas).
+
+```bash
+python -m vidmap.run_for_benchmark \
+  --dataset hilti \
+  --frontend-conf calib/panorama --mapping-conf calib/base
+```
+
+Evaluation uses scale-aligned WATE-AUC on view 0.
+
+</details>
+
 <details>
 <summary>[Results - click to expand]</summary>
 
@@ -539,6 +642,10 @@ better), evaluated at GT-supported estimated poses:
 | :--- | ---: | ---: | ---: | ---: | ---: | ---: |
 | LaMAR (sample-300, 50 sequences) | 0.878 | 92.43 | 93.08 | 91.79 | 91.03 | 90.41 |
 | CroCoDL (64 sequences) | 0.233 | 95.72 | 97.27 | 97.39 | 96.56 | 98.21 |
+| LaMAR (complete, 53 evaluated sequences) | 1.452 | 92.42 | 92.83 | 92.38 | 92.59 | 89.76 |
+| Oxford Spires (rectified monocular, 10 sequences) | 0.130 | 98.98 | 99.10 | 99.11 | 99.10 | 99.54 |
+| Oxford Spires (three-camera rig, 13 sequences) | Pending | Pending | Pending | Pending | Pending | Pending |
+| Hilti (three-view, 30 sequences) | Pending | Pending | Pending | Pending | Pending | Pending |
 
 Pose AUC, averaged equally across sequences (%, higher is better):
 

pyproject.toml

@@ -38,6 +38,9 @@ dependencies = [
   "geocalib @ git+https://github.com/cvg/GeoCalib.git@982f625d822fbb2b50c65d123fd1362a5c01ec2c",
 ]
 
+[project.optional-dependencies]
+dataset-preparation = ["rosbags>=0.10,<0.12"]
+
 [tool.scikit-build]
 minimum-version = "build-system.requires"
 cmake.version = ">=3.28"

vidmap/configs/frontend/calib/stereo.yaml

@@ -0,0 +1,7 @@
+defaults:
+  - rig
+
+rig:
+  keyframe_camera: 0
+  match_same_timestamp: [[0, 1]]
+  flag_same_timestamp_as_sequential: [[0, 1]]

vidmap/datasets/assets/benchmark_suites.json

@@ -0,0 +1,61 @@
+{
+  "spires": {
+    "scenes": [
+      "2024-03-12-keble-college-02",
+      "2024-03-12-keble-college-03",
+      "2024-03-12-keble-college-04",
+      "2024-03-12-keble-college-05",
+      "2024-03-13-observatory-quarter-01",
+      "2024-03-13-observatory-quarter-02",
+      "2024-03-14-blenheim-palace-01",
+      "2024-03-14-blenheim-palace-02",
+      "2024-03-14-blenheim-palace-05",
+      "2024-03-18-christ-church-02",
+      "2024-03-18-christ-church-03",
+      "2024-03-20-christ-church-05",
+      "2024-05-20-bodleian-library-02"
+    ],
+    "modes": {
+      "undistorted-mono": {"variant": "rectified", "num_cameras": 1},
+      "fisheye-mono": {"variant": "native", "num_cameras": 1},
+      "undistorted-rig": {"variant": "rectified", "num_cameras": 3}
+    }
+  },
+  "hilti": {
+    "scenes": [
+      "floor_1_2025-05-05_run_1",
+      "floor_1_2025-07-07_run_1",
+      "floor_1_2025-12-02_run_1",
+      "floor_2_2025-05-05_run_1",
+      "floor_2_2025-10-28_run_1",
+      "floor_2_2025-10-28_run_2",
+      "floor_2_2025-12-02_run_1",
+      "floor_2_2025-12-03_run_1",
+      "floor_3_2025-05-19_run_1",
+      "floor_3_2025-12-02_run_1",
+      "floor_4_2025-05-19_run_1",
+      "floor_4_2025-12-02_run_1",
+      "floor_5_2025-12-02_run_1",
+      "floor_6_2025-06-18_run_1",
+      "floor_6_2025-07-07_run_1",
+      "floor_6_2025-12-02_run_1",
+      "floor_6_2025-12-02_run_2",
+      "floor_7_2025-12-02_run_1",
+      "floor_7_2025-12-02_run_2",
+      "floor_7_2025-12-03_run_1",
+      "floor_EG_2025-10-16_run_1",
+      "floor_EG_2025-12-02_run_1",
+      "floor_EG_2025-12-02_run_2",
+      "floor_UG1_2025-05-19_run_1",
+      "floor_UG1_2025-06-18_run_1",
+      "floor_UG1_2025-10-16_run_1",
+      "floor_UG1_2025-12-02_run_1",
+      "floor_UG1_2025-12-02_run_2",
+      "floor_UG1_2025-12-03_run_1",
+      "floor_UG2_2025-12-02_run_1"
+    ],
+    "modes": {
+      "panorama": {"variant": "panorama", "num_cameras": 1}
+    }
+  }
+}

vidmap/datasets/assets/eth3d_slam/stereo_archives.tsv

@@ -0,0 +1,56 @@
+scene	split	size	http_etag	sha256
+cables_1	training	468306334	"""5cfba8ec-1be9c99e"""	62e9dec9d43f68703cd68207d9042eaaf135028a485618e03cad1c0db165cc77
+cables_2	training	37573372	"""5cfba99d-23d52fc"""	2b73a4754ff4638fd6ef7f6b707770a6957fa9a3b80a79404eb870fbda9dc41e
+cables_3	training	117435264	"""5cfba9b0-6ffeb80"""	48ce2abfe828bf5ed8262831fa62832652426c4ccb0aa6d6ffbae38be14ad526
+camera_shake_1	training	106974159	"""5cfba1e2-6604bcf"""	a99161a4edc5f7d96ab76322e3c75854bb7ff138edd0d5c029fecea99648cef9
+camera_shake_2	training	286022211	"""5cfba218-110c5a43"""	f72c8242245fabc059d85794fa6f30b7aeb8a8d8738a3d61d5de3c14ae2d9287
+camera_shake_3	training	91536560	"""5cfba288-574bcb0"""	acc0b20e33a68107d56cfcabed0bcd759a614a6739f7d302405878b5feb05df1
+ceiling_1	training	566799271	"""5cfba9f7-21c8aba7"""	951cd627b51397feaf455d44609c746a04186c49152d23e6a8dd5d323865d22d
+ceiling_2	training	560256988	"""5cfbaae3-2164d7dc"""	be1f2e07d8f6b19483a223302369ad1ca3355b36da7dae5bb7625af011b21c94
+desk_3	training	810709135	"""5cfbcba2-3052708f"""	72064c72412832e74ef41a594dc15d0dfbca363ed403ded1a819fa6c5c125b88
+desk_changing_1	training	1447857687	"""5cfbcd30-564c8e17"""	fe952331593677a1f76912dd7c00d897e8460610c8d0a1ab7154a14476558ff2
+einstein_1	training	179509025	"""5cfbb47e-ab31721"""	c264d228f3b757c75b14cf5ac3c8cf2308ca87d5d265a8c7112c2bc3d3a1c2d6
+einstein_2	training	528939870	"""5cfbb4e4-1f86fb5e"""	1b58e596fa636f057fca3a8f83a9bfebfa0a0a322d63f5cccadcb735a872966c
+einstein_flashlight	training	593871147	"""5cfbb6ce-2365c12b"""	168ccd4fd6032ff8f1754a760d9449bca3eff10a736f7b922d95cf32ce625bca
+einstein_global_light_changes_1	training	252793489	"""5cfbb7ba-f115291"""	61ed864d2f60cec06b671a4c4895b7478a7f6d685b96c861d95efaaa5d72d9c1
+einstein_global_light_changes_2	training	175055894	"""5cfbb820-a6f2416"""	82423843b9c77b39af980b1915c553937353c3775fcfc294584e3bb409b44736
+einstein_global_light_changes_3	training	788101180	"""5cfbb88d-2ef9783c"""	d726bfbb60637f38b3fa319c2cf95a7705f17f73b2b6e6e1f323c5aa3712e54b
+kidnap_1	training	322666172	"""5cfbc213-133b7ebc"""	95fe4edc61916405db40e7bf06b32211828e765d572a35b8bccecf277f8b6b2d
+large_loop_1	training	497813405	"""5cfba3c0-1dac079d"""	a8d3061b6bcdbee5044811400bcac9398cfb1636fd9a6f38d0e6145317e904ed
+mannequin_1	training	230855524	"""5cfba7e2-dc29364"""	d4e4e6f92b1d1ceb9982068f4abfb428b8a3e58b7259b68392b9ee92bb05a4a5
+mannequin_3	training	215029137	"""5cfba5b9-cd11591"""	c918d453c7bc095c74f211bb76538f0f324fabcb67491d99a10af8d36e738712
+mannequin_4	training	185035148	"""5cfba614-b07698c"""	491665d75e3283bbde1f3efa7533d2b439ec4649339330977f265e53a397d8d1
+mannequin_5	training	297932879	"""5cfba66b-11c2184f"""	bba6d1f9bba1e6a2ee38b1848b19aa3faa8a291e08cb760239d91ec806556144
+mannequin_7	training	223401150	"""5cfba842-d50d4be"""	9a2da8300fb898650ddbcc7e23793ae59c391c17e48e2a647c08b4bd0c990cbe
+mannequin_face_1	training	145937658	"""5cfba73e-8b2d4fa"""	00738e4900fec180d50f0597f4c97c9352925fc9fd719c33636c69dad46eded2
+mannequin_face_2	training	160830114	"""5cfba89d-99612a2"""	71faa4c8551da547429d944f5378cd464fd313a2456021592941f5e4ffbc8c2d
+mannequin_face_3	training	209410079	"""5cfba6e7-c7b581f"""	34f695285e04d53a217609714713235f4d636db93b5c83586c190af5b3083b18
+mannequin_head	training	232104066	"""5cfba782-dd5a082"""	cc9c8d5256be577c3fa4f734389dac8e879425b042b7361f3fff716163fbf200
+motion_1	training	837644219	"""5cfbb9de-31ed6fbb"""	55989272c1536370c107b43c770ba9b921d4440fade81060e114f96c2f8348da
+planar_2	training	309081903	"""5cfba488-126c372f"""	38bcbb0cf452a796349983839dbf6ffe8f0aa59ab6df8823b7cb3f9e1323eb4c
+planar_3	training	426904075	"""5cfba50d-19720a0b"""	b263db944759673817d8e1317ac79af93130f9c6cf3e96e708a932c0be78b466
+plant_1	training	23623541	"""5cfbb2e1-1687775"""	b66d68a9e232eede4148fde0a91b1caaac32c446a4a7a61f858c49127e69424b
+plant_2	training	45636091	"""5cfbb2ec-2b859fb"""	2f7fc740849f70571118ca7df41b6232bb01fa6414c4feafa44334f6738a1ec9
+plant_3	training	67715213	"""5cfbb301-409408d"""	336cedb4143ac8d9371b47281c24e0b915bb34e01be1aa5cc99cf1bf1b372ad7
+plant_4	training	32432886	"""5cfbb31c-1eee2f6"""	38f7dc4592b26d7322412d23c21242ffcf2c703f9afadfedbe425d6ccdea4e0f
+plant_5	training	62725140	"""5cfbb32b-3bd1c14"""	b6cdf6ed69fb0eb9e699fbb886a339c8dfd38cdf8ad4a9a3c8ee8df306a5240d
+plant_scene_1	training	251665904	"""5cfbb351-f001df0"""	26e0579c8e895e825c8585acb2be02632c1740fd091fdc8cc067a59418d724e5
+plant_scene_2	training	239439708	"""5cfbb3bc-e458f5c"""	8a7d4daae1360513803da98b8c5eff88179fe68fd820bf6739d282bedc52f07a
+plant_scene_3	training	212738765	"""5cfbb423-cae22cd"""	600469371dcdf382e732092779244917bf3a27a2a4ad55da49c2781752836af7
+reflective_1	training	1645498742	"""5cfbaf6f-62145176"""	a3f1600855394ed600bc2e11b3238c20095a97db28617d5c3a8b8e4c9b80db4f
+repetitive	training	690482274	"""5cfba0d9-2927ec62"""	187ecea76afd6a549d40b1f8016ca91d6c3d36ec22e464eb851fc9838ee01e5e
+sfm_bench	training	258391106	"""5cfbd28a-f66bc42"""	ffa1ac63401c479d53f6745b146508881961a7c9189f1ddd0a17aab4513d6e36
+sfm_garden	training	355485017	"""5cfbd18c-15304559"""	fb87f4dfd325852d9788123e344b37a8198298612ae63b493aa339ae7f8ea24e
+sfm_house_loop	training	254016693	"""5cfbd21d-f23fcb5"""	c8780810ece5d1402862f62abf4d7e7264445b790c2007fdd8f1619b6b01ffcf
+sfm_lab_room_1	training	94129054	"""5cfbd2f1-59c4b9e"""	188fa66dfc35b3d23619009c7be482f77d72434f97d874ce62498b1ccefbe70e
+sfm_lab_room_2	training	39590086	"""5cfbd318-25c18c6"""	ffc483753d34d2297a32db39a40383fdba559ee68679526b97bc97e1bcfe1e10
+sofa_1	training	342654754	"""5cfb9b01-146c7f22"""	f2de223f81f995cd715a4d465e485772fb24b862b4f2b7d7a61228f37de2b491
+sofa_2	training	242122750	"""5cfb9b93-e6e7ffe"""	2c9773f36de6baeaa911154692071897e37fbb1e956197d4e7435b19517ba67a
+sofa_3	training	187236617	"""5cfb9bfb-b290109"""	9bad0e06b38d4bd95cc5fbc80be17872b3ec07f0a39c511a61b8c752cce07add
+sofa_4	training	276821209	"""5cfb9c53-107ff4d9"""	fa5a2758db5f9cec3576ec074c07f363e381860065d5d4d9abea32d98575547c
+sofa_shake	training	176146324	"""5cfb9cca-a7fc794"""	02dac10b0a3ea83ba2c097f4d4b83e73bd517c0e05ed7cd6a95a4fc7b417d738
+table_3	training	449825872	"""5cfb97a5-1acfcc50"""	4f370e6df65f98ea80c3092a423164abac0549b8e0b528fade6927eef7f39fec
+table_4	training	372468096	"""5cfb985f-16336980"""	235dcc288a1c338f9db8606e25f87427800aae9a44b6913fada822526d9a0dbf
+table_7	training	600047762	"""5cfba2c4-23c40092"""	75f58be72b79150e61ec83c47c9331b028bc1c835c69d6aa566d8c05b25a328a
+vicon_light_1	training	298455960	"""5cfbabc1-11ca1398"""	8ea79f2536969cd07e83af72bd8615fb67d1131f995cee985e04ca19a72b72f5
+vicon_light_2	training	183749995	"""5cfbac39-af3cd6b"""	3a3437546448893e514c5666bda11d2b0cc5a1c9a9ad8e3c17d1e218a43fc723