Preprocessing Events to Tensors
Mostly for ML algorithms, but also some Model-based ones, it can be interesting to build tensors from events. These tensors can take the form of a time surface, event cube, histogram or differential frame. Each serves a different purpose, but all allow us to represent events into a frame-like data structure which can be fed to traditional or AI processing pipelines.
Preprocessing events with the C++ API
The Metavision C++ API provides preprocessing classes to build automatically those different types of tensors, with C++
bindings providing Python equivalent. The different tensor types are described in the
Knowledge Center
and the Legacy Python Event Preprocessing Tutorial.
The classes described below provide both a C++ API and Python bindings to these C++ implementations, which implies that
there is consistency between both languages, while the legacy version is only available in pure Python programming.
Also, the following API is more flexible to use, as it allows us to process events into any structure, as
Tensors can hold the data pointer of another structure’s data.
All in all, we recommend using the classes described below for event preprocessing.
These are available in the Core module of the SDK, through the preprocessors folder:
The DiffProcessor and
HardwareDiffProcessor follow the same mathematics but the first one
produces a float representation of the differential frame, while the second one follows the hardware Differential
frame generation of the GenX320 sensor, with in particular integer values manipulation. The same applies to Histogram
generation.
You can use the EventPreprocessorFactory to create the desired
instance, by providing a map with the desired preprocessing parameters, as well as the event input shape (typically
defined by the sensor used). The required parameters are described in the
EventPreprocessorFactory function documentation.
The way events are provided also needs to be specified in the create function as a template parameter. It will define
the type of input expected by the process_events method which
is provided by the EventPreprocessor interface (raw pointer to EventCD,
constant iterator to EventCD, etc).
std::unordered_map<std::string, Metavision::PreprocessingParameters> preprocess_map;
// Fill the map with desired parameters, here for
preprocess_map["type"] = Metavision::EventPreprocessorType::HARDWARE_HISTO;
preprocess_map["neg_saturation"] = (std::uint8_t)255;
preprocess_map["pos_saturation"] = (std::uint8_t)255;
// Instantiate the preprocessor
auto evt_input_shape = Metavision::TensorShape({{"H", height}, {"W", width}, {"C", 2}});
auto event_preprocessor = Metavision::EventPreprocessorFactory::create<const Metavision::EventCD *>(
preprocess_maps, evt_input_shape);
Then, you can use the process_events method to
apply the preprocessing of a batch of events to a pre-defined Tensor. This
Tensor can either allocate itself the memory of the tensor data, as shown on the first example below,
or be used as a wrapper to another structure’s data (particularly useful to avoid data copy) as shown in
the second example.
// Example 1: Standalone tensor
Tensor processed_data(evt_input_shape, Metavision::BaseType::UINT8);
event_preprocessor_->process_events(ref_time, events_vector_.data(),
events_vector_.data() + events_vector_.size(), processed_data);
// Example 2: Tensor wrapped around a cv::Mat data
cv::Mat allocated_matrix(rows, cols, CV_8UC1);
bool copy = false;
Tensor processed_data(evt_input_shape, Metavision::BaseType::UINT8,
static_cast<std::byte*>(allocated_matrix.data()), copy);
event_preprocessor_->process_events(ref_time, events_vector_.data(),
events_vector_.data() + events_vector_.size(), processed_data);
In fact, this tensor can represent any type of ‘tensor-like’ data: a histogram, a differential frame, an event-cube, etc, which makes it a generic data structure suitable for a variety of needs, including ML pipelines.
Finally, you can use this data in a dedicated algorithm, ML model, display, etc. For example, the ML module
Model class takes a map of tensors as input, which you can fill with such
preprocessors. The interface of the Tensor class also allows you to get a
pointer to the tensor’s data, and wrap it around your own data structure if needed, with cv::Mat for instance.
cv::Mat display_mat(height, width, CV_8UC2, processed_data.data<std::uint8_t>())
You can also directly create your own preprocessor if you don’t need to go through the factory function.
Metavision::HardwareHistoProcessor<const Metavision::EventCD *> event_preprocessor(width, height, 255, 255);
Preprocessing events with the Python API
There are two ways in the Python API to preprocess events.
The first method involves using the bindings of the C++ classes mentioned above, which are bound through the
EventPreprocessor class. See an example with the
HardwareHistoProcessor below.
from metavision_sdk_core import EventPreprocessor
event_preprocessor = EventPreprocessor.create_HardwareHistoProcessor(input_event_width=1280,
input_event_height=720,
neg_saturation=255,
pos_saturation=255)
All C++ preprocessing classes are available through these bindings with factory methods of this Python class:
A numpy array can be provided in place of the C++ tensor.
An example applied to the TimeSurface use-case can be found in the source code of
this sample
Another way to preprocess events in Python is through the legacy Python implementation of preprocessors. Examples can be found here.
Depending on the desired application and usage, the first or second one may be more relevant. If you are working on a Python PoC, which you will then want to port to C++, the Python bindings should be preferred, as they will provide same results as the C++ classes given same inputs, which is not guaranteed if the Python implementation is used. Also the bindings to the C++ classes should be more efficient: when processing an important amount of data, it might bring some efficiency gains to your pipeline.