Skip to main content
Version: Next

Generate VQAs

Bench2Drive-VL integrates DriveCommenter to generate VQAs from static dataset or CARLA runtime. DriveCommenter is a rule-based model modified from the open-source annotation script of DriveLM-CARLA dataset. Different from DriveLM-CARLA annotating script, DriveCommenter

  • Eliminates reliance on expert model decision information.
  • Optimizes problem structure by merging simple object-wise tasks into holistic problems, which helps reduce token usage.
  • Incorporates objects outside the front-facing camera's field of view, which is especially important in scenarios such as overtaking or yielding to emergency vehicles.
  • Expands the range of tasks to include environmental perception, high-level driving decision-making, and recognition of a broader set of traffic signs.
  • Handles out-of-distribution (OOD) cases in non-expert routes to ensure the model can react to all sorts of situations in CARLA simulator.

DriveCommenter Structure

If you want to generate a VQA dataset from CARLA using DriveCommenter, set environment variable MINIMAL=0 to run closed-loop inference without VLMs can do this job. Please refer to closed-loop inference.

If you want to know all supported questions of DriveCommenter and their qids, please refer to qid reference.

If you want to annotate VQAs from a dataset which has the same format as Bench2Drive dataset, please continue.

Limitations

We do not recommend using DriveCommenter to annotate customized static dataset instead of annotating from CARLA directly.

Since DriveCommenter is originally designed to annotate VQAs from Bench2Drive dataset, some rules in it are only valid in Bench2Drive. If the dataset is not from CARLA, or not from supported scenarios in Bench2Drive, the result is not accurate.

Write a startup script​

Place a startup script under ./B2DVL-Adapter:

B2DVL-Adapter/your_startup_script.sh
#!/bin/bash
export SUBSET=0 # generate from a subset of given dataset
export STRICT_MODE=1

export SUBSET_PATH=./subset_0.txt # subset file
export PROCESSED_PATH=./processed_paths_0.txt # checkpoint file
export CACHE_PATH=./.worker_0_cache

python ./drive_commenter_main.py --data-directory=/path/to/Bench2Drive/dataset --output-graph-directory=./outgraph     --path-maps=${CARLA_ROOT}/CarlaUE4/Content/Carla/Maps     --worker-count=1
Environment Variables
  • STRICT_MODE: DriveCommenter drives the ego vehicle after circumventing obstacles back to its original lane and handle more out of distribution situations if STRICT_MODE > 0.
  • CACHE_PATH: DriveCommenter supports dataset in .tar.gz. It will unzip some of the dataset temporarily in cache directory.
Multi-thread Issues

We do not recommend using multiple worker by --worker-count since multi-thread in python is not very good. You can run multiple DriveCommenter at the same time with different subset and checkpoint files to do the same.

Start generating​

Run the start up script.

cd ./B2DVL-Adapter
bash ./your_startup_script.sh