Note

This Python sample may be slow depending on the event rate of the scene and the configuration of the algorithm. We provide it to allow quick prototyping. For better performance, look at the corresponding C++ sample.

Sparse Optical Flow Sample using Python

The Python bindings of Metavision Computer Vision API can be used to compute the optical flow of objects moving in front of the camera. The optical flow is computed in a sparse way: flow information is generated on clusters of events and not for each event. Some other optical flow algorithms are described in the “Available Optical Flow Algorithms” section below.

The sample metavision_sparse_optical_flow.py shows how to use the python bindings of Metavision CV SDK to implement a pipeline for computing the sparse optical flow.

The source code of this sample can be found in <install-prefix>/share/metavision/sdk/cv/python_samples/metavision_sparse_optical_flow when installing Metavision SDK from installer or packages. For other deployment methods, check the page Path of Samples.

Expected Output

The sample visualizes events and the output optical flow with arrows indicating direction and magnitude of motion:

The sample can also generate a video with the output flow.

How to start

To start the sample based on recorded data, provide the full path to a RAW or HDF5 event file (here, we use a file from our Sample Recordings):

python metavision_sparse_optical_flow.py -i pedestrians.raw

Note

As explained in the Code Overview below, by default a filter algorithm (Metavision::SpatioTemporalContrastAlgorithmT) is applied to reduce the noise in the event stream. Depending on your input file, this might not be useful (or it could even suppress most of the events if another noise filter was already applied when recording the event file). To disable this filter, use the command line option --disable-stc

To check for additional options:

python metavision_sparse_optical_flow.py -h

Available Optical Flow Algorithms

This sample is using the Sparse Optical Flow algorithm, but the SDK API offers several other (dense) optical flow algorithms: Plane Fitting flow, Triplet Matching flow and Time Gradient flow. Those alternate algorithms are demonstrated in the Dense Flow C++ Sample.

The main differences between those flow algorithms are the following:

• Plane Fitting optical flow:

  • is based on plane-fitting in local neighborhood in time surface

  • is a simple and efficient algorithm, but run on all events hence is costly on high event-rate scenes

  • estimated flow is subject to noise and represents motion along edge normal (not full motion)

• Triplet Matching optical flow:

  • is based on finding aligned events triplets in local neighborhood

  • is a simple and very efficient algorithm, but run on all events hence is costly on high event-rate scenes

  • estimated flow is subject to noise and represents motion along edge normal (not full motion)

• Time Gradient optical flow:

  • is based on computing a spatio-temporal gradient on the local time surface using a fixed look-up pattern (i.e. it is essentially a simplified version of the Plane Fitting algorithm in which we only consider the pixels in a cross_shaped region (x0 +/- N, y0 +/- N) instead of a full NxN area around the pixel)

  • is a simple and very efficient algorithm, but run on all events hence is costly on high event-rate scenes

  • estimated flow is subject to noise and represents motion along edge normal (not full motion)

• Sparse optical flow:

  • is based on tracking of small edge-like features

  • is more complex but staged algorithm, leading to higher efficiency on high event-rate scenes

  • estimated flow represents actual motion, but requires fine tuning and compatible features in the scene

See also

To know more about those flow algorithms, you can check the paper about Normal Flow, the paper about Triplet Matching Flow and the patent about CCL Sparse Flow

Code Overview

Pipeline

Metavision Optical Flow sample implements the following pipeline:

Spatio Temporal Contrast Filter

First, if the option --disable-stc was not passed on the command line with the value True, then the filter algorithm metavision_sdk_cv.SpatioTemporalContrastAlgorithm is applied to reduce the noise in the event stream.

The filtered events are then sent to both the optical flow and frame generation stages.

Sparse Optical Flow

metavision_sdk_cv.SparseOpticalFlowAlgorithm is applied on the events stream and produces metavision_sdk_cv.EventOpticalFlow events for each internally detected events cluster. Events are clustered together based on their polarity and timestamp: clusters are formed from events connected spatially and temporally, and flow is computed for them.

For each incoming event, we search in a 3x3 neighborhood around its position for connection with the most recent events. The connectivity is decided based on the events polarities and timestamps. When a connection is made, we retrieve the cluster ID of the most recent event. If the input event connects to two or more events, we merge the different cluster IDs into a merged ID. We then label the current event with the selected ID or merged ID, or with a new cluster ID if no connection was found.

Illustration of the clustering process for one input event

Once the cluster ID has been determined, the event information are passed to a speed estimator specific for this cluster. The event’s position and timestamp are processed to update the cluster’s center and speed. We use a Luenberger state estimator to evaluate those outputs. A metavision_sdk_cv.EventOpticalFlow instance is created with the cluster ID, center and speed in pixel/s.

The parameters are preset for the sample. To see the full configuration possible, see metavision_sdk_cv.SparseOpticalFlowConfig.

Frame Generation

metavision_sdk_cv.SparseFlowFrameGeneratorAlgorithm generates a frame that will be used later on in the display stage to visualize the result of the metavision_sdk_cv.SparseOpticalFlowAlgorithm by rendering the estimated flows on the top of the events.

Display

When ready, the output frame is displayed on the screen:

Going further

Extract Speed Information

Within the numpy object all_flow_events (see source code), you can directly access speed vector components vx and vy of an event stream flow.

To do so, you can modify the Python source code to save all_flow_events object in a CSV file where columns are respectively (x, y, p, t, vx, vy, id, cx, cy):

• x - Column position of the event in the sensor

• y - Row position of the event in the sensor

• p - Polarity of the event

• t - Timestamp of the event (in us)

• vx - Speed in X axis in pixels per second in the image plane of the considered event

• vy - Speed in Y axis in pixels per second in the image plane of the considered event

• id - Feature ID of the considered event

• cx - Horizontal coordinate of the center of the feature used to compute speed of the considered event

• cy - Vertical coordinate of the center of the feature used to compute speed of the considered event

Speed information vx and vy is expressed in pixel/s.

cx and cy are not always computed for all optical flow algorithms. For example, they are computed in sparse optical flow algorithm and set to zero in Triplet Matching optical flow algorithm.

First import the python csv module:

import csv

Then replace this section in the source code:

if args.output_sparse_npy_filename:
    print("Writing output file: ", args.output_sparse_npy_filename)
    all_flow_events = np.concatenate(all_flow_events)
    np.save(args.output_sparse_npy_filename, all_flow_events)

With this one:

if args.output_sparse_npy_filename:
    print("Writing output file: ", args.output_sparse_npy_filename)
    all_flow_events = np.concatenate(all_flow_events)
    # np.save(args.output_sparse_npy_filename, all_flow_events)
    # save flow events in a csv file
    with open(args.output_sparse_npy_filename + '.csv', 'w', newline='') as file:
        writer = csv.writer(file)
        writer.writerows(all_flow_events)

Finally, call the sample with the --output-sparse-npy-filename option:

python metavision_sparse_optical_flow.py -i pedestrians.raw --output-sparse-npy-filename all_flow_events

This will create the file all_flow_events.csv with all the columns described above.