TL;DR: We show that high benchmark scores in end-to-end driving can be achieved without observing the current traffic scene, using solely memory from previous drives.
We introduce MemoryDrivoR, an analytical baseline for assessing whether current autonomous-driving benchmarks genuinely require models to interact with dynamic objects. Using MemoryDrivoR, we critically examine the NAVSIM benchmark and demonstrate that strong performance can be achieved using only static scene information. Although such information is necessary for autonomous driving, it should not be sufficient on its own. Therefore, MemoryDrivoR is intended solely as an auditing and research tool and is not designed for production use.
Here, we present the companion code for our paper "Driving on Memory" by Christian Löwens et al. The code allows the users to reproduce and extend the NAVSIM and Bench2Drive results reported in the study. Please cite this work when reporting, reproducing or extending our results. This software is a research prototype, solely developed for and published as part of the paper. It will neither be maintained nor monitored in any way.
Released checkpoints and memory banks can be used directly for evaluation, while the training and memory-bank construction docs explain how to regenerate the corresponding artifacts from scratch.
navsim1/is used for all NAVSIM training and fine-tuning, including the NAVSIMv2 models, and for NAVSIMv1 evaluation.navsim2/is used only for NAVSIMv2 evaluation.bench2drive/contains the MemoryDrivoR Bench2Drive overlay used for training, memory-bank construction, and closed-loop evaluation.
NAVSIM:
- Create local folders, install environments, and download NAVSIM artifacts or data with docs/navsim/setup.md.
- Evaluate released NAVSIM artifacts or regenerate them with docs/navsim/reproduction.md.
- Reproduce NAVSIM paper ablations with docs/navsim/ablations.md.
Bench2Drive:
- Create local folders, install the environment and CARLA, and download Bench2Drive artifacts or data with docs/bench2drive/setup.md.
- Evaluate released Bench2Drive artifacts or regenerate them with docs/bench2drive/reproduction.md.
- Reproduce Bench2Drive geographic split ablations with docs/bench2drive/ablations.md.
Large artifacts are not committed to this repository. Download them into the local paths shown below. The setup guides contain copy-paste download commands.
| Version | Artifact | Local path | Size |
|---|---|---|---|
| NAVSIMv1 | Original DrivoR checkpoint | weights/original_ckpts/base_drivor_navsim1.pth |
291 MB |
| NAVSIMv2 | Original* DrivoR checkpoint | weights/original_ckpts/base_drivor_navsim2.pth |
291 MB |
| NAVSIMv2 | Reproduced* DrivoR checkpoint | weights/original_ckpts/base_drivor_navsim2.pth |
291 MB |
| Bench2Drive | Reproduced DrivoR checkpoint | weights/original_ckpts/base_drivor_b2d.ckpt |
295.4 MB |
*Consistent with reports from others, we were unable to reproduce the NAVSIMv2 performance reported for DrivoR. To ensure a consistent and fair evaluation, we therefore used our independently reproduced checkpoint in all experiments (incl. memory building and fine-tuning for the models below).
| Version | Artifact | Local path | Size |
|---|---|---|---|
| NAVSIMv1 | MemoryDrivoR checkpoint | weights/memorydrivor_navsim1.ckpt |
305.8 MB |
| NAVSIMv1 | Ego-only checkpoint | weights/ego_only_navsim1.ckpt |
269.8 MB |
| NAVSIMv1 | HD-map-only checkpoint | weights/hdmap_only_navsim1.ckpt |
197.9 MB |
| NAVSIMv2 | MemoryDrivoR checkpoint | weights/memorydrivor_navsim2.ckpt |
305.8 MB |
| NAVSIMv2 | Ego-only checkpoint | weights/ego_only_navsim2.ckpt |
269.8 MB |
| NAVSIMv2 | HD-map-only checkpoint | weights/hdmap_only_navsim2.ckpt |
197.9 MB |
| Bench2Drive | MemoryDrivoR checkpoint | weights/memorydrivor_b2d.ckpt |
310.1 MB |
| Bench2Drive | Ego-only checkpoint | weights/ego_only_b2d.ckpt |
273.2 MB |
| Version | Artifact | Local path | Size |
|---|---|---|---|
| NAVSIMv1 | Memory bank | memory_banks/memory_bank_navsim1.pt |
6.3 GB |
| NAVSIMv2 | Memory bank | memory_banks/memory_bank_navsim2.pt |
6.3 GB |
| Bench2Drive | Memory bank | memory_banks/memory_bank_b2d.pt |
22.2 GB |
The external DINOv2 backbone is downloaded from Hugging Face during setup.
Except where otherwise noted, MemoryDrivoR's original source code and modifications are licensed under AGPL-3.0. See the LICENSE file for details.
The documentation asset assets/ego_prediction.gif is derived from the nuPlan dataset and is expressly excluded from the AGPL-3.0 license. It is subject to CC BY-NC-SA 4.0 and the nuPlan/Motional Dataset Terms. These asset may be used only for non-commercial purposes.
For a list of third-party software and other licensed materials included in MemoryDrivoR, see 3rd-party-licenses.txt.