Unify benchmark preparation and loading

README.md

@@ -361,7 +361,7 @@ python -m vidmap.datasets.prepare --dataset "$DATASET"
 ```
 
 This downloads and validates the supported evaluation inventory, then converts
-it into the canonical `images/`, `rec/`, and testset layout. LaMAR intentionally
+it into the dataset layouts below. LaMAR intentionally
 prepares only the images referenced by its benchmark testset: sessions and
 images absent from that testset are not retained. Downloads resume after
 interruption and completed sequences are reused. Validated source archives are
@@ -384,48 +384,65 @@ Download sizes are the checksum-pinned source archives; prepared sizes cover
 the canonical dataset after conversion. LaMAR archives are session-level, so
 the download includes unused frames inside each required session ZIP; only the
 94,489 testset images are retained. Retaining downloads requires roughly the sum
-of both columns.
+of both columns, plus conversion staging.
 
 After preparation, the selected dataset is available under
 `./local/datasets/<dataset>/`, with its benchmark definitions under
 `./local/testsets/<dataset>/`:
 
 ```text
 local/
-├── datasets/
-│   ├── lamar/
-│   │   └── scene: <CAB|HGE|LIN>/
-│   │       ├── images/
-│   │       │   └── cam0/<capture>/<timestamp>.jpg  # e.g. ios_2022-06-30_15.55.53/130156492755.jpg
-│   │       ├── rec/
-│   │       └── preparation.json
-│   ├── euroc/
-│   │   └── scene: <machine_hall-MH_01_easy|machine_hall-MH_02_easy|...>/
-│   │       ├── images/
-│   │       │   └── cam0/<timestamp>.png  # e.g. 1403636580863555584.png
-│   │       ├── rec/
-│   │       └── preparation.json
-│   ├── eth3d_slam/
-│   │   └── scene: <plant_1|plant_2|...>/
-│   │       ├── images/
-│   │       │   └── cam0/<timestamp>.<ext>  # e.g. 5865.976533.png
-│   │       ├── rec/
-│   │       ├── unposed_images.json
-│   │       └── preparation.json
-│   └── crocodl/  # Unlike LaMAR, CroCoDL has GT per session, not per scene.
-│       └── <location>/  # e.g. ARCHE_D2
-│           └── <session>/  # e.g. ios_2023-07-12_20.46.14_000
-│               ├── images/
-│               │   └── cam0/<timestamp>.jpg  # e.g. 4971459477.jpg
-│               ├── rec/
-│               └── metadata.json
-└── testsets/
-    ├── lamar/<scene>/<mode>.yaml  # e.g. HGE/sample.yaml
-    ├── euroc/<sequence>/all.yaml  # e.g. machine_hall-MH_01_easy/all.yaml
-    ├── eth3d_slam/<sequence>/all.yaml  # e.g. plant_1/all.yaml
-    └── crocodl/<ios-location>/<mode>.yaml  # e.g. ios-ARCHE_D2/sample-100.yaml
+├── datasets/<dataset>/<scene>/<recording>/
+│   ├── preparation.json
+│   └── <variant>/
+│       ├── images/cam0/...         # cam1/, cam2/, ... when available
+│       └── rec/                   # Matching calibration and reference poses
+└── testsets/<dataset>/<scene>/<mode>.yaml
 ```
 
+| Dataset | Variant(s) |
+| --- | --- |
+| LaMAR, ETH3D-SLAM, CroCoDL | `native` |
+| EuRoC | `rectified` |
+
+<details>
+<summary>[Examples for each dataset - click to expand]</summary>
+
+```text
+local/datasets/
+├── lamar/CAB/ios_2021-06-02_14.21.49_000/
+│   ├── preparation.json
+│   └── native/
+│       ├── images/cam0/...
+│       └── rec/
+├── euroc/machine_hall/MH_01_easy/
+│   ├── preparation.json
+│   └── rectified/
+│       ├── images/cam0/...
+│       └── rec/
+├── eth3d_slam/plant_1/0/
+│   ├── preparation.json
+│   └── native/
+│       ├── images/cam0/...
+│       └── rec/
+└── crocodl/ios-ARCHE_D2/ios_2023-07-12_20.46.14_000/
+    ├── preparation.json
+    └── native/
+        ├── images/cam0/...
+        └── rec/
+
+local/testsets/
+├── lamar/CAB/sample.yaml
+├── euroc/machine_hall/all.yaml
+├── eth3d_slam/plant_1/all.yaml
+└── crocodl/ios-ARCHE_D2/complete.yaml  # Also sample-*.yaml
+```
+
+</details>
+
+Each `rec/` contains calibration and camera poses, without 3D points.
+Select a benchmark with `--mode`; its testset specifies the recording, image variant and image IDs.
+
 </details>
 
 ### Run and evaluate

docs/cameras.md

@@ -89,22 +89,25 @@ In this example:
 
 ## Panoramas
 
-Put your 360° equirectangular images in `/data/images/cam0/`.
+Put your 360° equirectangular images in `/data/images/cam0/`, or pass a stitched
+MP4/MOV panorama as `--input_data`.
 Save `vidmap/configs/frontend/my_panorama.yaml`:
 
 ```yaml
 defaults: [calib/rig]
 rig:
   projection:
-    image_size: [840, 840]  # width, height
+    aspect_ratio: [1, 1]  # width : height
     horizontal_fov_degrees: 120.0
     views:
       0: {yaw_degrees: 0.0, pitch_degrees: 0.0}
       1: {yaw_degrees: 120.0, pitch_degrees: 0.0}
       2: {yaw_degrees: 240.0, pitch_degrees: 0.0}
 ```
 
-**Full cube:** keep square images and replace the FOV and views inside `projection`:
+View resolution follows the tracking settings (`tracks.images`).
+
+**Full cube:** keep `[1, 1]` and replace the FOV and views inside `projection`:
 
 ```yaml
 horizontal_fov_degrees: 90.0

docs/files.md

@@ -10,7 +10,7 @@ e.g. `cam0/1.png`.
 | Uncalibrated monocular | A directory containing `cam0/`, or an MP4/MOV video |
 | Calibrated monocular | A directory containing `cam0/` |
 | Physical rig | A directory containing `cam0/`, `cam1/`, … |
-| Panorama | A directory containing `cam0/` with 360° equirectangular images |
+| Panorama | A directory containing `cam0/` with 360° equirectangular images, or a stitched MP4/MOV panorama |
 
 Uncalibrated monocular images use natural filename order (`frame2.png` before
 `frame10.png`). Rigs, calibrated monocular and panoramas resolve numeric filenames

vidmap/benchmark/cli.py

@@ -330,7 +330,7 @@ def main(argv=None):
         validate_profile_selection(selection, enabled=run_options.profile)
     except ValueError as error:
         parser.error(str(error))
-    selection.dataset_spec.prepare(conf.selection.scene)
+    selection.dataset_spec.prepare(conf.selection.scene, mode=conf.selection.mode)
 
     import time
 

vidmap/benchmark/report.py

@@ -197,7 +197,6 @@ def collect(
             if evaluation_options is not None:
                 conf = dataclasses.replace(conf, evaluation=evaluation_options)
                 wate_auc_full_thresholds = evaluation_options.windowed_ate_auc_full_thresholds
-            dataset.prepare()
             result_evaluator = BenchmarkResultEvaluator(
                 conf,
                 dataset,
@@ -249,7 +248,9 @@ def collect(
                             wate_colls[mode][scene][wkey + "_trajs"] = len(all_vals)
                         else:
                             wate_colls[mode].setdefault(scene, {})[wkey] = float(np.mean(all_vals))
-        if getattr(args, "scene", None) is None:
+        if dataset.name == "euroc":
+            scenes = list(out)
+        elif getattr(args, "scene", None) is None:
             scenes = result_evaluator.selection.dataset_layout.scenes
         else:
             scenes = args.scene

vidmap/benchmark/result_evaluation.py

@@ -5,10 +5,13 @@
 
 import matplotlib
 import pycolmap
+import yaml
 from tqdm import tqdm
 
-from vidmap.benchmark.trajectory import AbsoluteTrajectoryEvaluator, AbsoluteTrajectoryOptions
+from vidmap.benchmark.trajectory import AbsoluteTrajectoryEvaluator
+from vidmap.configuration.dump import frontend_config_path
 from vidmap.datasets.selection import PreparedTargetSelection
+from vidmap.frontend.options.rig import RigOptions
 from vidmap.run_options import RunOptions
 from vidmap.utils.logging import log_context, progress_bars_enabled
 
@@ -40,39 +43,18 @@ def __init__(
         self.output_name = output_name
         self.run_options = RunOptions() if run_options is None else run_options
         self.selection = PreparedTargetSelection(conf.selection, dataset)
-        self.evaluator = AbsoluteTrajectoryEvaluator(options=AbsoluteTrajectoryOptions.from_config(conf.evaluation))
+        rig = None if self.frontend_conf is None else self.frontend_conf.pipeline.rig
+        self.evaluator = AbsoluteTrajectoryEvaluator.from_config(conf, rig)
 
-    @staticmethod
-    def remap_image_ids(reconstruction, ground_truth):
-        """Return a reconstruction whose image IDs match ground truth names."""
-        ground_truth_ids = {image.name: image_id for image_id, image in ground_truth.images.items()}
-        id_mapping = {
-            image_id: ground_truth_ids[image.name]
-            for image_id, image in reconstruction.images.items()
-            if image.name in ground_truth_ids and image_id != ground_truth_ids[image.name]
-        }
-        if not id_mapping:
-            return reconstruction
-
-        remapped = pycolmap.Reconstruction()
-        for camera in reconstruction.cameras.values():
-            remapped.add_camera_with_trivial_rig(camera)
-        for image_id, image in reconstruction.images.items():
-            remapped_image = pycolmap.Image(
-                image_id=id_mapping.get(image_id, image_id),
-                camera_id=image.camera_id,
-                name=image.name,
-            )
-            if image.has_pose:
-                remapped.add_image_with_trivial_frame(remapped_image, image.cam_from_world())
-            else:
-                remapped.add_image_with_trivial_frame(remapped_image)
-        for point3d in reconstruction.points3D.values():
-            remapped.add_point3D(point3d)
-        return remapped
+    def _report_group(self, case):
+        # EuRoC reports individual recordings, independently of their storage scene.
+        if self.selection.dataset_spec.name == "euroc":
+            return f"{case.scene}-{case.recording}"
+        return case.scene
 
     def evaluate_case(self, case, scene_parser, outputs, recalls, wates):
         """Load, evaluate, and aggregate one persisted benchmark reconstruction."""
+        group = self._report_group(case)
         try:
             results = BenchmarkResultStore(
                 self.conf,
@@ -84,8 +66,15 @@ def evaluate_case(self, case, scene_parser, outputs, recalls, wates):
                 output_name=self.output_name,
             )
             reconstruction = pycolmap.Reconstruction(results.output_dir / "rec")
-            reconstruction = self.remap_image_ids(reconstruction, scene_parser.rec)
-            evaluation = self.evaluator.evaluate(
+            evaluator = self.evaluator
+            frontend_path = frontend_config_path(results.output_dir)
+            if frontend_path.is_file():
+                frontend = yaml.safe_load(frontend_path.read_text())
+                rig = frontend["rig"] if "rig" in frontend else None
+                evaluator = AbsoluteTrajectoryEvaluator.from_config(
+                    self.conf, None if rig is None else RigOptions(**rig)
+                )
+            evaluation = evaluator.evaluate(
                 estimated_reconstruction=reconstruction,
                 ground_truth_reconstruction=scene_parser.rec,
             )
@@ -97,29 +86,33 @@ def evaluate_case(self, case, scene_parser, outputs, recalls, wates):
             )
 
             summary = evaluation.summarize(verbose=logger.isEnabledFor(logging.INFO))
-            outputs[case.scene][case.testset_type].append(summary["all"])
+            outputs[group][case.testset_type].append(summary["all"])
             if "recall" in summary:
-                recalls[case.scene][case.testset_type].append(summary["recall"])
+                recalls[group][case.testset_type].append(summary["recall"])
             for key in (key for key in summary if key.startswith("wate_") or key.startswith("wrre_")):
-                target = wates[case.scene][key][case.testset_type]
+                target = wates[group][key][case.testset_type]
                 if key.startswith("wate_auc_errors_"):
                     target.extend(summary[key])
                 else:
                     target.append(summary[key])
-            return True, case.scene
+            return True, group
         except Exception as error:
             if self.run_options.terminate_on_error:
                 raise error
             logger.exception("Error processing %s: %s", case.scene, error)
-            return False, case.scene
+            return False, group
 
     def evaluate_saved(self):
         """Evaluate every selected reconstruction saved on disk."""
-        scenes = self.selection.dataset_layout.scenes
-        outputs = {scene: defaultdict(list) for scene in scenes}
-        recalls = {scene: defaultdict(list) for scene in scenes}
-        wates = {scene: defaultdict(lambda: defaultdict(list)) for scene in scenes}
-        counts = {scene: {"completed": 0, "failed": 0} for scene in scenes}
+        groups = self.selection.dataset_layout.scenes
+        if self.selection.dataset_spec.name == "euroc":
+            groups = tuple(
+                self._report_group(case) for _, cases in self.selection.iter_scene_targets() for case in cases
+            )
+        outputs = {group: defaultdict(list) for group in groups}
+        recalls = {group: defaultdict(list) for group in groups}
+        wates = {group: defaultdict(lambda: defaultdict(list)) for group in groups}
+        counts = {group: {"completed": 0, "failed": 0} for group in groups}
         for case, scene_parser, _ in tqdm(
             self.selection.prepared_targets(),
             disable=not progress_bars_enabled(),
@@ -131,6 +124,6 @@ def evaluate_saved(self):
                 testset=case.testset_desc,
                 config=self.conf.name,
             ):
-                ok, scene = self.evaluate_case(case, scene_parser, outputs, recalls, wates)
-                counts[scene]["completed" if ok else "failed"] += 1
+                ok, group = self.evaluate_case(case, scene_parser, outputs, recalls, wates)
+                counts[group]["completed" if ok else "failed"] += 1
         return outputs, counts, recalls, wates