Evaluation Configurations

Evaluation module's configurations.

Example

eval_config.json

{
    "EVAL_SUBSET": true, // eval a subset of given infer result folder
    "USE_CHECKPOINT": false, // use a file to record evaluation process
    "SUBSET_FILE": "./eval_configs/subset.txt", // subset file
    "CHECKPOINT_FILE": "./eval_configs/finished_scenarios.txt", // checkpoint file
    "INFERENCE_RESULT_DIR": "./infer_results", // path to inference results
    // when doing closed-loop inference, this dir is ./output/infer_results/model_name+input_mode
    "B2D_DIR": "/path/to/Bench2Drive/dataset", // evaluation script uses annotations in b2d,
    // when doing closed-loop inference, this dir is ./eval_v1(SAVE_PATH you specified)/model_name+input_mode
    "ORIGINAL_VQA_DIR": "../Carla_Chain_QA/carla_vqa_gen/vqa_dataset/outgraph",
    // when doing closed-loop inference, this dir is ./output/vqagen/model_name+input_mode
    "FRAME_PER_SEC": 10, // sensor fps
    "LOOK_FUTURE": false // not used now, to be removed
}

Fields

• INFER_SUBSET (type: bool): Whether inference a subset of given infer result folder or not.

• USE_CHECKPOINT (type: bool): Whether use a checkpoint file to record the process of inference, so that B2DVL only inference scenarios which are not in the checkpoint file.

• SUBSET_FILE (type: str): Path to the subset file. You can leave blank if not used.

• CHECKPOINT_FILE (type: str): Path to the checkpoint file. You can leave blank if not used.

• INFERENCE_RESULT_DIR (type: str): Path to inference results folder. When doing closed-loop inference, this dir is ./output/infer_results/model_name+input_mode.

• B2D_DIR (type: str): Path to original Bench2Drive dataset. Evaluation script uses annotations in B2D. When doing closed-loop inference, this dir is ${SAVE_PATH}/model_name+input_mode.

• ORIGINAL_VQA_DIR (type: str): Path to the VQAs generated by DriveCommenter. When doing closed-loop inference, this dir is ./output/vqagen/model_name+input_mode.

• FRAME_PER_SEC (type: int): The sampling rate of the sensors.

• LOOK_FUTURE (type: bool): Not used now, to be removed.

subset and checkpoint

The grammar of subset file and checkpoint file is same as one in open-loop inference.