SDK ML Detection And Tracking API

Generates frames from boxes and tracks

metavision_ml.detection_tracking.display_frame.draw_box_events(frame, box_events, label_map, force_color=None, draw_score=True, thickness=1, color_from='class_id', confidence_field='class_confidence', dic_history={})

Draws boxes on a RGB image.

  • frame (np.ndarray) – H, W, 3 image of dtype uint8

  • box_events (np.ndarray) – Box_events in EventBbox format.

  • label_map (list) – List of class names indexed by class_id.

  • force_color (int list) – If not None, overrides the color from color_field and is put as the color

  • all boxes. (of) –

  • draw_score (bool) – Whether to add confidence to the boxes.

  • thickness (int) – Rectangle line thickness.

  • color_from (string) – Field of the EventBbox used to choose the box color, defaults to class_id.

  • confidence_field (string) – Field of the EventBbox used to read confidence score.

  • dic_history (dict) – Dictionary where keys are track_id and values are np.arrays of boxes


Drawn image.

Return type


metavision_ml.detection_tracking.display_frame.draw_detections_and_tracklets(ts, frame, width, height, detections, tracklets, label_map={0: 'background', 1: 'pedestrian', 2: 'two wheeler', 3: 'car'}, list_previous_tracks=[])

Draws a visualization for both detected boxes and tracked boxes side by side.

Detections are on the left pane, with box colors indicating class membership. Tracks are drawn on the right pane with colors denoting track ids.

  • ts (int) – Current timestamp in us. (Used for display only).

  • frame (np.ndarray) – Array of dtype np.uint8 and of shape height x width * 2 x 3

  • width (int) – Width of the imager in pixels.

  • height (int) – Height of the imager in pixels.

  • detections (nd.array) – Array of EventBbox to be drawn.

  • tracklets (ndarray) – Array of EventTrackedBox to be drawn.

  • label_map (dict) – Dictionary mapping class ids to the name of the corresponding class

  • list_previous_tracks (list) – list of np.arrays of tracks (one np.array for each previous timestep)

Detection Loader which reads boxes from a CSV file generated by the C++ detection and tracking pipeline

This function reloads a text file produced by the c++ detection and tracking pipeline and converts it into a dictionary of numpy structured arrays of EventBbox


detections_filename (str) – Filename of detections in the same format as the C++ detection and tracking pipeline

It returns a python dict where the keys are the timestamps. For each timestamp, the associated value is a numpy array of EventBbox for that particular timestamp


>>> dic_ts_boxes = detections_csv_loader("detections.txt")
>>> np_array_of_all_boxes_detected_at_timestamp_1000 = dic_ts_boxes[1000]
class metavision_ml.detection_tracking.object_detector.ObjectDetector(directory, events_input_width, events_input_height, runtime='cpu', network_input_width=None, network_input_height=None)

Class handling the inference of object detection algorithm.

  • directory (string) – Path to a folder containing a model.ptjit network file and a info_ssd_jit.json containing a few hyperparameters.

  • events_input_width (int) – Width of the event sensor used, in pixels.

  • events_input_height (int) – Height of the event sensor used, in pixels.

  • runtime (string) – Corresponds to the torch.device used for computation (“cpu”, “cuda:0”, etc.)

  • network_input_width (int) – Width of the tensor used by the network as input, can be lower than events_input_width to reduce computation.

  • network_input_height (int) – Height of the tensor used by the network as input, can be lower than events_input_height to reduce computation.


Object computing tensor representation from the input events.




Minimal confidence value for a box to be considered.




Minimal Intersection Over Union (IOU) value for a box to be discarded by non maximum suppression.




Object handling Non maximum suppression.




Returns a dict which contains the labels of the classes


Gets number of output classes (including background class)

process(ts, frame_buffer_np)

Pass the input frame through the object detector and return the obtained box events.

  • ts (int) – Current timestamp in us.

  • frame_buffer_np (np.ndarray) – Input frame buffer.


Resets the memory cells of the neural network