Note

This Python sample may be slow depending on the event rate of the scene and the configuration of the algorithm. We provide it to allow quick prototyping. For better performance, look at the corresponding C++ sample.

Generic Tracking using Python

The Python bindings of Metavision SDK Analytics provide the metavision_sdk_analytics.TrackingAlgorithm class for generic object tracking.

The sample metavision_generic_tracking.py shows how to use metavision_sdk_analytics.TrackingAlgorithm to track objects.

Note that the Analytics API also provides a lighter implementation of metavision_sdk_analytics.TrackingAlgorithm restricted to non-colliding objects: metavision_sdk_analytics.SpatterTrackerAlgorithm demonstrated in the Spatter Tracking sample.

The source code of this sample can be found in <install-prefix>/share/metavision/sdk/analytics/python_samples/metavision_generic_tracking when installing Metavision SDK from installer or packages. For other deployment methods, check the page Path of Samples.

Expected Output

Metavision Generic Tracking sample visualizes all events and the tracked objects by showing a bounding box around each tracked object with an ID of the tracked object shown next to the bounding box:

Setup & requirements

By default, Metavision Generic Tracking looks for objects of size between 10x10 an 300x300 pixels. Use the command line options --min-size and --max-size to adapt the sample to your scene.

How to start

To start the sample based on the live stream from your camera, run:

Linux

python3 metavision_generic_tracking.py

Windows

python metavision_generic_tracking.py

To start the sample based on recorded data, provide the full path to a RAW file (here, we use a file from our Sample Recordings):

Linux

python3 metavision_generic_tracking.py -i traffic_monitoring.raw

Windows

python metavision_generic_tracking.py -i traffic_monitoring.raw

To check for additional options:

Linux

python3 metavision_generic_tracking.py -h

Windows

python metavision_generic_tracking.py -h

Code Overview

The sample implements the following pipeline:

Note

The pipeline also allows to apply filters on the events using the algorithms ActivityNoiseFilterAlgorithm and TrailFilterAlgorithm that can be enabled and configured using the activity-time-ths and --activity-trail-ths command line options. Other software filter could be used like SpatioTemporalContrastAlgorithm. Alternatively, to filter out events, you could enable some of the ESP blocks of your sensors.

The tracking algorithm consumes CD events and produces tracking results (i.e. metavision_sdk_analytics.EventTrackingDataBuffer). Those tracking results contain the bounding boxes with unique IDs of tracked objects.

The algorithm is synchronous meaning that it will try to detect and track objects at every call. However, the tracking results may be affected by the amount of data passed during the call. That’s why the RollingEventCDBuffer and the EventsIterator classes are both used in the sample to de-correlate the tracking frequency from the accumulation time.

Indeed, the RollingEventCDBuffer class first allows for creating overlapping time-slices of events by implementing a rolling window over an event stream. Two strategies are possible to define the rolling window width: every N events or every N us. The second strategy is used in this sample:

# Rolling event buffer
buffer_config = RollingEventBufferConfig.make_n_us(args.accumulation_time_us)
rolling_buffer = RollingEventCDBuffer(buffer_config)

In addition to that, the EventsIterator class is used to control the frequency at which the algorithm is called. To do so, the iterator is configured to slice the events at the expected frequency:

# Events iterator on Camera or event file
delta_t = int(1000000 / args.update_frequency)
mv_iterator = EventsIterator(input_path=args.event_file_path, start_ts=args.process_from,
                             max_duration=args.process_to - args.process_from if args.process_to else None,
                             delta_t=delta_t, mode="delta_t")

The sliced events are first filtered and then processed:

# Process events
for evs in mv_iterator:
    # Dispatch system events to the window
    EventLoop.poll_and_dispatch()

    # Process events
    if args.activity_time_ths > 0:
        activity_noise_filter.process_events(evs, events_buf)
        if args.activity_trail_ths > 0:
            trail_filter.process_events_(events_buf)

        process_tracking(events_buf.numpy())

    elif args.activity_trail_ths > 0:
        trail_filter.process_events(evs, events_buf)

        process_tracking(events_buf.numpy())
    else:
        process_tracking(evs)

    if window.should_close():
        break

The processing includes pushing the events in the rolling buffer, calling the tracking algorithm on the accumulated events and drawing the results:

def process_tracking(evs):
    if len(evs) != 0:
        rolling_buffer.insert_events(evs)
        tracking_algo.process_events(rolling_buffer, tracking_results)
        BaseFrameGenerationAlgorithm.generate_frame(rolling_buffer, output_img)
        draw_tracking_results(evs['t'][-1], tracking_results, output_img)

    window.show_async(output_img)
    if args.out_video:
        video_writer.write(output_img)

The following image shows an example of output:

Expected Output from Metavision Tracking Sample

Note

Both the tracking frequency and the accumulation time are configurable with command line parameters

Tracking Algorithm

The generic tracking algorithm consists of 4 main parts:

Cluster making
Data association
Tracker initialization
Tracking

The tracking algorithm can be configured via metavision_sdk_analytics.TrackingConfig.

1. Cluster making

In metavision_sdk_analytics.TrackingConfig.ClusterMaker, clusters are built from input events.

Two clustering methods are implemented:

metavision_sdk_analytics.TrackingConfig.ClusterMaker.SimpleGrid - builds clusters based on a regular grid (default):
- The camera FOV is divided in elementary cells using a regular grid; the size of the cells is defined by cell_width x cell_height.
- For each cell, the number of events in the cell for a given time-slice is compared to the activation-threshold, if it exceeds the threshold then the cell is considered as active
- Active cells are connected into clusters
metavision_sdk_analytics.TrackingConfig.ClusterMaker.MedoidShift - builds clusters from events based on spatial and temporal distances between neighboring events. If the spatial and temporal distances between the event and its neighboring event are smaller than metavision_sdk_analytics.TrackingConfig.medoid_shift_spatial_dist and metavision_sdk_analytics.TrackingConfig.medoid_shift_temporal_dist, then the event goes to the same cluster as its neighbour, otherwise, it creates a new cluster.

2. Data association

In metavision_sdk_analytics.TrackingConfig.DataAssociation, clusters are associated to trackers.

The following data association methods are implemented:

metavision_sdk_analytics.TrackingConfig.DataAssociation.NEAREST - associates a cluster to the nearest tracker
metavision_sdk_analytics.TrackingConfig.DataAssociation.IOU - associates a cluster to the tracker with the largest intersection area (default)

3. Tracker initialization

In the tracker initialization part, new trackers are initialized, and bounding box proposals are made from input clusters/events.

One of the following motion models (metavision_sdk_analytics.TrackingConfig.MotionModel) is used to predict the position of the tracker:

metavision_sdk_analytics.TrackingConfig.MotionModel.SIMPLE - assumes that the velocity is constant (default)
metavision_sdk_analytics.TrackingConfig.MotionModel.INSTANT - takes the last measured velocity
metavision_sdk_analytics.TrackingConfig.MotionModel.SMOOTH - models velocity as a smooth evolving quantity
metavision_sdk_analytics.TrackingConfig.MotionModel.KALMAN - models velocity in a mode Kalman

Note

Below is a code example to select the motion model SMOOTH:

# Tracking Algorithm
tracking_config = TrackingConfig()  # Default configuration
tracking_config.motion_model = TrackingConfig.MotionModel.Smooth

4. Tracking

Two types of trackers metavision_sdk_analytics.TrackingConfig.Tracker are implemented:

metavision_sdk_analytics.TrackingConfig.Tracker.ELLIPSE - based on an event by event update of the pose and the shape of the tracker. The tracker is represented as a Gaussian, and the update is performed by weighting each event contribution with metavision_sdk_analytics.TrackingConfig.EllipseUpdateFunction and updating the tracker’s pose/size using metavision_sdk_analytics.TrackingConfig.EllipseUpdateMethod.
metavision_sdk_analytics.TrackingConfig.Tracker.CLUSTERKF - Kalman tracker (default) uses the result of clustering as an observation state to predict the current state of the tracker.