Inference Pipeline of Optical Flow using Python

This Python script allows you to quickly set up an inference pipeline for Optical Flow Estimation.

You can use it with the pre-trained PyTorch flow model flow_model_alpha.ckpt from our pre-trained models.

The source code of this sample can be found in <install-prefix>/share/metavision/sdk/ml/python_samples/flow_inference when installing Metavision SDK from installer or packages. For other deployment methods, check the page Path of Samples.

Expected Output

The script takes events or preprocessed event features as input and outputs optical flow.

An example of the output is shown below:

Warning

Disclaimer: The provided models are neither expected to provide state-of-the-art results, neither supposed to be computationally efficient. They provide a basis for ML exploration with our event-based technology. In particular, with default parameters and depending on the sensor used, they might not run live with your camera and ML pipeline.

Setup & requirements

To run the script, you will need:

• a pre-trained PyTorch model (e.g. flow_model_alpha.ckpt from our pre-trained models)

• event data using one of these formats:

  • an event-based camera

  • an event file: RAW, DAT or HDF5 event file (you can find event files in our Sample Recordings page)

  • an HDF5 tensor files created with our generate_hdf5 script as explained in the tutorial Precomputing features as HDF5 tensor files.

Note

Since HDF5 tensor file contains preprocessed features, you need to be sure that the same preprocessing method is used for the flow model and the HDF5 file. For instance, our trained flow model uses event_cube method, so if you want to use HDF5 tensor files as input, they need to be processed with event_cube as well.

How to start

To start the script based on recorded data, you need to provide the full path to the input file and the path to the pre-trained model. For example:

python flow_inference.py /path/to/flow_model_alpha.ckpt pedestrians.raw

Note

1. To read directly from a camera, provide the camera serial number if there are several cameras otherwise leave it blank (passing an empty string)

2. Use -w /path/to/output.mp4 to generate a mp4 video

3. Use --mode to specify your visualization mode:

   • arrows (default mode): the size of the arrow represents the flow magnitude, while the color of the flow represents the angle

   • sharp : use this mode to check the quality of flow estimation, if the input features contain micro time bins, in other words, if the preprocessing method is event cube or multi_channel timesurface. This will give sharper edges when the flow direction is correctly modelled.

Warning

To make the best use of the model, it is important to set the accumulation time interval (--delta-t) correctly in corresponding to the speed motion of the object. It is especially important for objects with high-speed motions: a long accumulation time would produce erroneous results.

To find the full list of options, run:

python flow_inference.py -h