Note
This Python sample may be slow depending on the event rate of the scene and the configuration of the algorithm. We provide it to allow quick prototyping. For better performance, look at the corresponding C++ sample.
Sparse Optical Flow Sample using Python
The Python bindings of Metavision Computer Vision API can be used to compute the optical flow of objects moving in front of the camera. The optical flow is computed in a sparse way: flow information is generated on clusters of events and not for each event. Some other optical flow algorithms are described in the “Available Optical Flow Algorithms” section below.
The sample metavision_sparse_optical_flow.py shows how to use the python bindings of Metavision CV SDK to implement
a pipeline for computing the sparse optical flow.
The source code of this sample can be found in <install-prefix>/share/metavision/sdk/cv/python_samples/metavision_sparse_optical_flow
when installing Metavision SDK from installer or packages. For other deployment methods, check the page
Path of Samples.
Expected Output
The sample visualizes events and the output optical flow with arrows indicating direction and magnitude of motion:
The sample can also generate a video with the output flow.
How to start
To start the sample based on recorded data, provide the full path to a RAW or HDF5 event file (here, we use a file from our Sample Recordings):
Linux
python3 metavision_sparse_optical_flow.py -i pedestrians.raw
Windows
python metavision_sparse_optical_flow.py -i pedestrians.raw
Note
As explained in the :ref` Code Overview <chapter_samples_cv_sparse_flow_cpp_code>` below, by default
a filter algorithm (Metavision::SpatioTemporalContrastAlgorithmT) is applied
to reduce the noise in the event stream. Depending on your input file, this might not be useful
(or it could even suppress most of the events if another noise filter was already applied when
recording the event file). To disable this filter, use the command line option --disable-stc
To check for additional options:
Linux
python3 metavision_sparse_optical_flow.py -h
Windows
python metavision_sparse_optical_flow.py -h
Available Optical Flow Algorithms
This sample is using the Sparse Optical Flow algorithm, but the SDK API offers several other (dense) optical flow algorithms: Plane Fitting flow, Triplet Matching flow and Time Gradient flow. Those alternate algorithms are demonstrated in the Dense Flow C++ Sample.
The main differences between those flow algorithms are the following:
Plane Fitting optical flow:
is based on plane-fitting in local neighborhood in time surface
is a simple and efficient algorithm, but run on all events hence is costly on high event-rate scenes
estimated flow is subject to noise and represents motion along edge normal (not full motion)
Triplet Matching optical flow:
is based on finding aligned events triplets in local neighborhood
is a simple and very efficient algorithm, but run on all events hence is costly on high event-rate scenes
estimated flow is subject to noise and represents motion along edge normal (not full motion)
Time Gradient optical flow:
is based on computing a spatio-temporal gradient on the local time surface using a fixed look-up pattern (i.e. it is essentially a simplified version of the Plane Fitting algorithm in which we only consider the pixels in a cross_shaped region (x0 +/- N, y0 +/- N) instead of a full NxN area around the pixel)
is a simple and very efficient algorithm, but run on all events hence is costly on high event-rate scenes
estimated flow is subject to noise and represents motion along edge normal (not full motion)
Sparse optical flow:
is based on tracking of small edge-like features
is more complex but staged algorithm, leading to higher efficiency on high event-rate scenes
estimated flow represents actual motion, but requires fine tuning and compatible features in the scene
See also
To know more about those flow algorithms, you can check the paper about Normal Flow, the paper about Triplet Matching Flow and the patent about CCL Sparse Flow
Code Overview
Spatio Temporal Contrast Filter
First, if the option --disable-stc was not passed on the command line with the value True,
then the filter algorithm metavision_sdk_cv.SpatioTemporalContrastAlgorithm is applied
to reduce the noise in the event stream.
The filtered events are then sent to both the optical flow and frame generation stages.
Sparse Optical Flow
metavision_sdk_cv.SparseOpticalFlowAlgorithm is applied on the events stream and produces
metavision_sdk_cv.EventOpticalFlow events for each internally detected events cluster. Events are clustered
together based on their polarity and timestamp: clusters are formed from events connected spatially and temporally, and flow is
computed for them.
For each incoming event, we search in a 3x3 neighborhood around its position for connection with the most recent events. The connectivity is decided based on the events polarities and timestamps. When a connection is made, we retrieve the cluster ID of the most recent event. If the input event connects to two or more events, we merge the different cluster IDs into a merged ID. We then label the current event with the selected ID or merged ID, or with a new cluster ID if no connection was found.
Illustration of the clustering process for one input event
Once the cluster ID has been determined, the event information are passed to a speed estimator specific for this cluster.
The event’s position and timestamp are processed to update the cluster’s center and speed.
We use a Luenberger state estimator to evaluate those outputs. A metavision_sdk_cv.EventOpticalFlow instance is created with the cluster ID,
center and speed in pixel/s.
The parameters are preset for the sample. To see the full configuration possible, see metavision_sdk_cv.SparseOpticalFlowConfig.
Frame Generation
metavision_sdk_cv.SparseFlowFrameGeneratorAlgorithm generates a frame that will be used later on
in the display stage to visualize the result of the metavision_sdk_cv.SparseOpticalFlowAlgorithm by rendering the
estimated flows on the top of the events.
Going further
Extract Speed Information
Within the numpy object all_flow_events (see source code), you can directly access speed vector components vx and vy of an event stream flow.
To do so, you can modify the Python source code to save all_flow_events object in a CSV file where columns are respectively (x, y, p, t, vx, vy, id, cx, cy):
x - Column position of the event in the sensor
y - Row position of the event in the sensor
p - Polarity of the event
t - Timestamp of the event (in us)
vx - Speed in X axis in pixels per second in the image plane of the considered event
vy - Speed in Y axis in pixels per second in the image plane of the considered event
id - Feature ID of the considered event
cx - Horizontal coordinate of the center of the feature used to compute speed of the considered event
cy - Vertical coordinate of the center of the feature used to compute speed of the considered event
Speed information vx and vy is expressed in pixel/s.
First import the python csv module:
import csv
Then replace this section in the source code:
if args.output_sparse_npy_filename:
print("Writing output file: ", args.output_sparse_npy_filename)
all_flow_events = np.concatenate(all_flow_events)
np.save(args.output_sparse_npy_filename, all_flow_events)
With this one:
if args.output_sparse_npy_filename:
print("Writing output file: ", args.output_sparse_npy_filename)
all_flow_events = np.concatenate(all_flow_events)
# np.save(args.output_sparse_npy_filename, all_flow_events)
# save flow events in a csv file
with open(args.output_sparse_npy_filename + '.csv', 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(all_flow_events)
Finally, call the sample with the --output-sparse-npy-filename option:
Linux
python3 metavision_sparse_optical_flow.py -i pedestrians.raw --output-sparse-npy-filename all_flow_events
Windows
python metavision_sparse_optical_flow.py -i pedestrians.raw --output-sparse-npy-filename all_flow_events
This will create the file all_flow_events.csv with all the columns described above.