Evaluation of Detection and Tracking Result with COCO KPIs

This Python script allows you to evaluate COCO detection KPIs on the output formats of our detection and tracking pipelines (see our C++ sample and our Python sample).

The source code of this sample can be found in <install-prefix>/share/metavision/sdk/ml/python_samples/eval_coco_kpi when installing Metavision SDK from installer or packages. For other deployment methods, check the page Path of Samples.

Expected Output

COCO KPIs: mean Average Precision (mAP) and mean Average Recall (mAR) and their variants.

Setup & requirements

To run the script, you will need to provide the following input and parameters:

  • the ground-truth directory (--GT-dir): a directory which contains Ground-truth (GT) Bbox encoded with custom Numpy EventBbox format.

  • the ground-truth dictionary (--GT-labels-dict): a JSON dictionary in the form of {Class ID: Class Description}, e.g. {“0”: “pedestrian”, “1”, “car”}.

  • the model directory (--DET-model-dir): a directory which contains a JSON dictionary of the detection categories (used to filter out irrelevant categories from the GT), named info_ssd_jit.json.

  • the directory of detection Bbox (--DET-dir)

  • the type of detection and its output format: the supported types include csv_detections, csv_tracks, npy_detections, npy_tracks

  • the time interval between two consecutive detections: in µs

  • the time interval between two consecutive GT labels: in µs

Note that:

  1. All the detection categories need to be incorporated in the ground-truth.

  2. The type of detection supports both detection and tracking Bbox, in both NPY (output format of the Python sample) and CSV (output format of the C++ sample).

If you have already downloaded our test dataset during the test of Linux installation or test of Windows installation, you can find examples of input data in <OPENEB_SRC_DIR>/datasets/metavision_ml/detection/coco_kpi.

How to start

To run the script:

python eval_coco_kpi.py --GT-dir /path/to/gt --GT-labels-dict /path/to/label.json --DET-dir /path/to/det --DET-model-dir /path/to/model  --detections-delta-t <time interval of detection> --labels-delta-t <time interval of GT label> --detection-type <type of detection>

To find the full list of options, run:

python eval_coco_kpi.py -h