Get Started

In this Section, we will create a minimal sample to get started with Metavision SDK. The goal of this sample is to introduce in the simplest possible way some basic concepts of Metavision SDK and create a first running example.

Setup

As with the other samples presented in this guide, we use CMake for our building environment. Therefore, the first step is to create a CMakeLists.txt file to compile our sample.

Create a new text file, copy the following text, and save it as CMakeLists.txt:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
project(metavision_sdk_get_started)

cmake_minimum_required(VERSION 3.5)

set(CMAKE_CXX_STANDARD 14)

find_package(MetavisionSDK COMPONENTS core driver REQUIRED)

set (sample metavision_sdk_get_started)
add_executable(${sample} ${sample}.cpp)
target_link_libraries(${sample} MetavisionSDK::core MetavisionSDK::driver)

The first interesting line in this file is the following:

7
find_package(MetavisionSDK COMPONENTS core driver REQUIRED)

This line indicates CMake that it should include two COMPONENTS from the Metavision SDK: core and driver. These two components are required for any basic operations involving cameras and events processing. We will see in the following sections how.

The second interesting line in this file is the following:

11
target_link_libraries(${sample} MetavisionSDK::core MetavisionSDK::driver)

This line indicates CMake that it should link our sample against Metavision::core and Metavision::driver.

The other lines are standard CMake instructions. For more information, have a look at the CMake documentation.

Now that the CMake setup is completed, you can create the CPP code file.

Create a file, name it metavision_sdk_get_started.cpp and save it next to your CMakeLists.txt file.

Note

The file name must be the same as the one specified in the CMakeLists.txt file created before on line 9. The executable file will have the same name.

Start the camera

The first operation we want to do is open the event-based camera or a pre-recorded file. In Metavision SDK, live cameras and pre-recorded RAW files are managed in the same way: by using the Metavision::Camera class. Let’s see how.

Copy the following code in the CPP file created in the previous section:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#include <metavision/sdk/driver/camera.h>

// main loop
int main(int argc, char *argv[]) {
    Metavision::Camera cam; // create the camera

    if (argc >= 2) {
        // if we passed a file path, open it
        cam = Metavision::Camera::from_file(argv[1]);
    } else {
        // open the first available camera
        cam = Metavision::Camera::from_first_available();
    }

    // start the camera
    cam.start();

    // keep running while the camera is on or the video is not finished
    while (cam.is_running()) {
        std::cout << "Camera is running!" << std::endl;
    }

    // the video is finished, stop the camera.
    // Note: we will never get here with a live camera
    cam.stop();
}

It is now possible to compile your code and execute it.

You can compile your sample following the same procedure as with any other Metavision sample:

mkdir build && cd build
cmake ..
cmake --build .

To execute it, you either need to plug in an event-based camera or use a pre-recorded RAW file. In the following, we will use a RAW file from our dataset, but you can use any available RAW file.

Execute the following command if you want to process data from a live camera:

Linux

./metavision_sdk_get_started

Windows

metavision_sdk_get_started.exe

Execute the following command if you want to process data from a pre-recorded RAW file (from Metavision Dataset in this example):

Linux

./metavision_sdk_get_started monitoring_40_50hz.raw

Windows

metavision_sdk_get_started.exe monitoring_40_50hz.raw

Note

We suggest that you run the samples in this section with a pre-recorded RAW file, as some functions are less easy to demonstrate with a live camera (as we will see).

If everything went fine, the output in your console should be similar to this:

...
Camera is running!
Camera is running!
Camera is running!
Camera is running!
Camera is running!
Camera is running!
Camera is running!
...

Note

The large numbers of console outputs in this and the following samples may slow down their execution. If this creates problems or prevents you from testing the code, we suggest you remove all std::cout lines to improve execution speed.

Let’s analyze the sample line by line:

1
#include <metavision/sdk/driver/camera.h>

This line includes the header for the Metavision::Camera class. The Camera class can be used to get the events from a live camera, as the name suggests, but also from a pre-recorded RAW file.

 5
 6
 7
 8
 9
10
11
12
13
    Metavision::Camera cam; // create the camera

    if (argc >= 2) {
        // if we passed a file path, open it
        cam = Metavision::Camera::from_file(argv[1]);
    } else {
        // open the first available camera
        cam = Metavision::Camera::from_first_available();
    }

Here we can see how, depending on the command line parameters, an instance of the Camera class can be used to read from a file (cam = Metavision::Camera::from_file();) or to stream from a camera (cam = Metavision::Camera::from_first_available();). From now on, the source code will be the same whether the events are coming from a file or from a live camera.

16
    cam.start();

This line simply starts the camera, from now on the camera will be active and stream events.

18
19
20
21
    // keep running while the camera is on or the video is not finished
    while (cam.is_running()) {
        std::cout << "Camera is running!" << std::endl;
    }

This while loop simply continues printing Camera is running! while the camera is running (cam.is_running()). For now, the camera runs until the recording is finished, in case of a pre-recorded RAW file, or forever in case of a live camera (at least, while the camera is plugged in).

25
    cam.stop();

This line stops the camera. Since a live camera will never get outside the while loop explained before, this point can be reached only when using a pre-recorded RAW file. We will see in the next sections how to stop a live camera.

To summarize, in this section, we learned how to open a camera or a file using the Metavision::Camera class, and how to start and stop a camera.

Get the events

In the previous section, we learned how to start and stop an instance of Metavision::Camera. This is a mandatory step, but not very useful by itself. In this section, we will learn how to get access to the events produced.

The Metavision::Camera class, as many other components of Metavision, works using callbacks. This means that we do not directly get access to the events. Instead, we need to create a function that will be called automatically whenever a buffer of events is ready to be processed.

Copy the following code into your metavision_sdk_get_started.cpp file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
#include <metavision/sdk/driver/camera.h>
#include <metavision/sdk/base/events/event_cd.h>

// this function will be associated to the camera callback
void count_events(const Metavision::EventCD *begin, const Metavision::EventCD *end) {
    int counter = 0;

    // this loop allows us to get access to each event received in this callback
    for (const Metavision::EventCD *ev = begin; ev != end; ++ev) {
        ++counter; // count each event

        // print each event
        std::cout << "Event received: coordinates (" << ev->x << ", " << ev->y << "), t: " << ev->t
                  << ", polarity: " << ev->p << std::endl;
    }

    // report
    std::cout << "There were " << counter << " events in this callback" << std::endl;
}

// main loop
int main(int argc, char *argv[]) {
    Metavision::Camera cam; // create the camera

    if (argc >= 2) {
        // if we passed a file path, open it
        cam = Metavision::Camera::from_file(argv[1]);
    } else {
        // open the first available camera
        cam = Metavision::Camera::from_first_available();
    }

    // add the event callback. This callback will be called periodically to provide access to the most recent events
    cam.cd().add_callback(count_events);

    // start the camera
    cam.start();

    // keep running while the camera is on or the video is not finished
    while (cam.is_running()) {}

    // the video is finished, stop the camera.
    // Note: we will never get here with a live camera
    cam.stop();
}

Recompile the file and execute it. The output should be similar to this:

...
Event received: coordinates (10, 252), t: 1638, polarity: 1
Event received: coordinates (12, 252), t: 1638, polarity: 1
Event received: coordinates (63, 252), t: 1638, polarity: 1
Event received: coordinates (67, 252), t: 1638, polarity: 1
There were 288 events in this callback
Event received: coordinates (66, 252), t: 1638, polarity: 1
Event received: coordinates (64, 252), t: 1638, polarity: 1
Event received: coordinates (65, 252), t: 1638, polarity: 1
...

If we look at the main loop, line 33, we can see the new call to setup the callback:

33
    cam.cd().add_callback(count_events);

This line indicates the Metavision::Camera class to call the function count_events when there are CD events available. The count_events function is defined at the top of the file (line 5).

Compared to the last example, here we removed the std::cout call in the while loop to avoid cluttering the console output.

Since we are now dealing directly with events, we need to include the relevant header.

2
#include <metavision/sdk/base/events/event_cd.h>

Let’s now have a look at the count_events function:

5
void count_events(const Metavision::EventCD *begin, const Metavision::EventCD *end) {

As explained, this function is called every time new CD events are available. To be compatible with the callback, a function needs to return void and have as parameters two iterators to the beginning and the end of the buffer of events that will be available to the function. These two iterators allow us to access all the new events passed in this callback.

We can now create a for loop to compute some simple statistics on the events and print them:

 9
10
    for (const Metavision::EventCD *ev = begin; ev != end; ++ev) {
        ++counter; // count each event

For each event we get, we increment a counter (line 10), allowing us to count how many events are available in the current callback.

We also print the details of each event, displaying its coordinates, timestamp and polarity:

13
14
15
        std::cout << "Event received: coordinates (" << ev->x << ", " << ev->y << "), t: " << ev->t
                  << ", polarity: " << ev->p << std::endl;
    }

We have now access to the events produced by the Metavision::Camera class. With this, we will be able to write all our event-based algorithms.

To summarize, in this section, we learned how to get access to the events and add a callback. We used this callback to count the number of events we receive and print their characteristics.

Add a dedicated processing class

In the previous section, we learned how to use the Camera callback to get access to the events. Creating an independent function for the callback is useful for simple analysis, but not practical for more advanced uses.

In this section, we will show how to create a dedicated class for event processing.

As before, copy the following code into your metavision_sdk_get_started.cpp file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
#include <metavision/sdk/driver/camera.h>
#include <metavision/sdk/base/events/event_cd.h>

// this class will be used to analyze the events
class EventAnalyzer {
public:
    // class variables to store global information
    int global_counter                 = 0; // this will track how many events we processed
    Metavision::timestamp global_max_t = 0; // this will track the highest timestamp we processed

    // this function will be associated to the camera callback
    // it is used to compute statistics on the received events
    void analyze_events(const Metavision::EventCD *begin, const Metavision::EventCD *end) {
        std::cout << "----- New callback! -----" << std::endl;

        // time analysis
        // Note: events are ordered by timestamp in the call back, so the first event will have the lowest timestamp and
        // the last event will have the highest timestamp
        Metavision::timestamp min_t = begin->t;     // get the timestamp of the first event of this callback
        Metavision::timestamp max_t = (end - 1)->t; // get the timestamp of the last event of this callback
        global_max_t = max_t; // events are ordered by timestamp, so the current last event has the highest timestamp

        // counting analysis
        int counter = 0;
        for (const Metavision::EventCD *ev = begin; ev != end; ++ev) {
            ++counter; // increasing local counter
        }
        global_counter += counter; // increase global counter

        // report
        std::cout << "There were " << counter << " events in this callback" << std::endl;
        std::cout << "There were " << global_counter << " total events up to now." << std::endl;
        std::cout << "The current callback included events from " << min_t << " up to " << max_t << " microseconds."
                  << std::endl;

        std::cout << "----- End of the callback! -----" << std::endl;
    }
};

// main loop
int main(int argc, char *argv[]) {
    Metavision::Camera cam;       // create the camera
    EventAnalyzer event_analyzer; // create the event analyzer

    if (argc >= 2) {
        // if we passed a file path, open it
        cam = Metavision::Camera::from_file(argv[1]);
    } else {
        // open the first available camera
        cam = Metavision::Camera::from_first_available();
    }

    // add the event callback. This callback will be called periodically to provide access to the most recent events
    cam.cd().add_callback([&event_analyzer](const Metavision::EventCD *ev_begin, const Metavision::EventCD *ev_end) {
        event_analyzer.analyze_events(ev_begin, ev_end);
    });

    // start the camera
    cam.start();

    // keep running while the camera is on or the video is not finished
    while (cam.is_running()) {}

    // the video is finished, stop the camera.
    // Note: we will never get here with a live camera
    cam.stop();

    // print the global statistics
    float length_in_seconds = event_analyzer.global_max_t / 1000000.0;
    std::cout << "There were " << event_analyzer.global_counter << " events in total." << std::endl;
    std::cout << "The total duration was " << length_in_seconds << " seconds." << std::endl;
    if (length_in_seconds >= 1) { // no need to print this statistics if the video was too short
        std::cout << "There were " << event_analyzer.global_counter / (event_analyzer.global_max_t / 1000000.0)
                  << " events per seconds on average." << std::endl;
    }
}

Recompile the file and execute it. The output should be similar to this:

...
----- New callback! -----
There were 288 events in this callback
There were 185048 total events up to now.
The current callback included events from 25855 up to 25870 microseconds.
----- End of the callback! -----
...

The main change of this new sample is that we replaced the event_count function with a dedicated class EventAnalyzer. However, the concept is the same: you need to define a function to add as a callback to the Metavision::Camera class. Before this function was defined independently, now it is a member function of the class EventAnalyzer.

Notice how we also replaced the simple callback of the previous sample with a more complex one:

54
55
56
    cam.cd().add_callback([&event_analyzer](const Metavision::EventCD *ev_begin, const Metavision::EventCD *ev_end) {
        event_analyzer.analyze_events(ev_begin, ev_end);
    });

Given that now the function we want to use for the callback is inside a class, it would have been necessary to create a wrapper function to manage the communication between the callback and the class function. This would have been possible, but a more concise solution is the use of a lambda expression. With this call, we can set a new callback without the need to explicitly create a separate function.

Let’s have a look now at the EventAnalyzer::analyze_events() function.

Similarly to the previous sample, the goal of this function is compute interesting statistics on the received events: the min and the max timestamps of the received events, and their count.

To extract the min and the max timestamps, we leverage the fact that the events in each callback are provided in temporal order, so we can assume that the first event has the min timestamp of this batch, and the last event has the max timestamp.

19
20
        Metavision::timestamp min_t = begin->t;     // get the timestamp of the first event of this callback
        Metavision::timestamp max_t = (end - 1)->t; // get the timestamp of the last event of this callback

As with the previous sample, we count how many events are in this callback, by increasing a counter with a for loop.

24
25
26
27
        int counter = 0;
        for (const Metavision::EventCD *ev = begin; ev != end; ++ev) {
            ++counter; // increasing local counter
        }

One of the advantages of using a class is that we can store information over multiple callbacks. For example, here we store the max timestamp over all callbacks:

21
        global_max_t = max_t; // events are ordered by timestamp, so the current last event has the highest timestamp

We also store the total number of events received:

28
        global_counter += counter; // increase global counter

Once our instance of Metavision::camera has been stopped (that is, when the pre-recorded RAW file finishes), we can print global statistics of this file:

68
69
70
71
72
73
74
75
    // print the global statistics
    float length_in_seconds = event_analyzer.global_max_t / 1000000.0;
    std::cout << "There were " << event_analyzer.global_counter << " events in total." << std::endl;
    std::cout << "The total duration was " << length_in_seconds << " seconds." << std::endl;
    if (length_in_seconds >= 1) { // no need to print this statistics if the video was too short
        std::cout << "There were " << event_analyzer.global_counter / (event_analyzer.global_max_t / 1000000.0)
                  << " events per seconds on average." << std::endl;
    }

To summarize, in this section, we learned how using a dedicated class we can do more complex analysis and store information between callbacks. We also introduced the use of lambda expressions to simplify the setup of a callback.

Add a display

In the previous sections, we learned how to use callbacks to get access to the events. With this knowledge, it is now possible to create a display to visualize the output of the camera.

Doing it from scratch, while possible, would be complex and long. Luckily, the Metavision SDK contains a class that can help us with the visualization: Metavision::CDFrameGenerator.

Note

Event-based cameras do not produce frames, but a stream of independent events. To visualize these events, we must artificially build a frame by accumulating events over time. This can be done in different ways, but the easiest is to create a binary frame: we start with a frame where each pixel is set to zero, and we set to one the corresponding pixel every time we receive an event. We need to choose the frequency at which these frames are constructed, that is, the equivalent FPS, and how long we accumulate the events, that is, the accumulation time. See also the documentation for our Metavision Player for more information.

First, add this snippet of code to the previous sample, before cam.start(), line 57:

    // get camera resolution
    int camera_width  = cam.geometry().width();
    int camera_height = cam.geometry().height();

    // create a frame generator for visualization
    // this will get the events from the callback and accumulate them in a cv::Mat
    Metavision::CDFrameGenerator cd_frame_generator(camera_width, camera_height);

    // this callback tells the camera to pass the events to the frame generator, who will then create the frame
    cam.cd().add_callback(
        [&cd_frame_generator](const Metavision::EventCD *ev_begin, const Metavision::EventCD *ev_end) {
            cd_frame_generator.add_events(ev_begin, ev_end);
        });

    int fps         = 25;               // event-cameras do not have a frame rate, but we need one for visualization
    float wait_time = 1.0 / fps * 1000; // how much we should wait between two frames
    cv::Mat cd_frame;                   // the cv::Mat where the events will be accumulated
    std::string window_name = "Metavision SDK Get Started";

    // this function is used to tell the frame generator what to do with the frame and how often to generate it
    cd_frame_generator.start(
        fps, [&cd_frame](const Metavision::timestamp &ts, const cv::Mat &frame) { frame.copyTo(cd_frame); });

    cv::namedWindow(window_name, CV_GUI_EXPANDED);
    cv::resizeWindow(window_name, camera_width, camera_height);

Now replace the while loop that was on line 62, with this code:

    // keep running while the camera is on or the video is not finished
    while (cam.is_running()) {
        // display the frame if it's not empty
        if (!cd_frame.empty()) {
            cv::imshow(window_name, cd_frame);
        }

        // if the user presses the `q` key, quit the loop
        if ((cv::waitKey(wait_time) & 0xff) == 'q') {
            break;
        }
    }

We also need to include the Metavision::CDFrameGenerator header. Copy this line at the beginning of your file:

#include "metavision/sdk/core/utils/cd_frame_generator.h"

You can find the complete sample in your installation directory:

Linux

/usr/share/metavision/sdk/core/samples/get_started/metavision_sdk_get_started.cpp

Windows

C:\Program Files\Prophesee\share\metavision\sdk\core\sample\get_started\metavision_sdk_get_started.cpp

Recompile and execute. This is the expected result:

../../_images/get_started_output.png

Let’s now see how the visualization works.

The frame generator works in the following way: we need to create it by passing the resolution of the camera. We then need to pass events to it for displaying. This is done as before by adding a callback to the Metavision::Camera class. Finally, we need to specify if we want to perform any additional processing on the generated frame, or simply copy it. Note that Metavision::CDFrameGenerator only creates a frame from events, it does not manage the visualization. For visualization, we make use of OpenCV functions.

First we need to get the camera resolution, to be able to properly initialize the frame visualization. We can use the Metavision::Camera::Geometry() facility to get the camera resolution:

60
61
    int camera_width  = cam.geometry().width();
    int camera_height = cam.geometry().height();

We can now create the frame generator:

65
    Metavision::CDFrameGenerator cd_frame_generator(camera_width, camera_height);

We then need to pass the events from the camera to the frame generator, so that it can accumulate them and create the frame for visualization. Similarly to what we saw before, this is done using another callback: every time some events are available, they will be passed to the frame generator.

Note

We can add multiple callbacks to the same camera by calling the Metavision::Camera::cd()::add_callback() function multiple times on the same instance. These multiple callbacks are not competing for events: each callback will be called with the same events once they become available.

The callback for the frame generator is added as before using a lambda expression:

79
80
81
82
    cam.cd().add_callback(
        [&cd_frame_generator](const Metavision::EventCD *ev_begin, const Metavision::EventCD *ev_end) {
            cd_frame_generator.add_events(ev_begin, ev_end);
        });

At this point, we need to start the frame generator. To do so, we need to specify a few details: the equivalent FPS for the displaying, the target OpenCV Mat, and the window name. The variable wait_time will be used later on, you can ignore it for now. The class Metavision::CDFrameGenerator also works via callbacks: when the generator is started, we need to pass a function that is used to process the created frame. This can be useful if we want to modify the frame before displaying it, for example, to add useful information. In our example, we simply copy the generated frame to our frame for display, using the copyTo function.

73
74
75
76
77
78
79
80
    int fps         = 25;               // event-cameras do not have a frame rate, but we need one for visualization
    float wait_time = 1.0 / fps * 1000; // how much we should wait between two frames
    cv::Mat cd_frame;                   // the cv::Mat where the events will be accumulated
    std::string window_name = "Metavision SDK Get Started";

    // this function is used to tell the frame generator what to do with the frame and how often to generate it
    cd_frame_generator.start(
        fps, [&cd_frame](const Metavision::timestamp &ts, const cv::Mat &frame) { frame.copyTo(cd_frame); });

Finally, we create the window and resize it:

82
83
    cv::namedWindow(window_name, CV_GUI_EXPANDED);
    cv::resizeWindow(window_name, camera_width, camera_height);

The frame generator now will generate a frame at a fixed frequency and copy it to our cd_frame variable. To display it, we can use the OpenCV imshow function. We need to do it periodically, every time a new frame is available. To do this, we need to modify the while loop in the following way:

88
89
90
91
92
93
94
95
96
97
98
99
    // keep running while the camera is on or the video is not finished
    while (cam.is_running()) {
        // display the frame if it's not empty
        if (!cd_frame.empty()) {
            cv::imshow(window_name, cd_frame);
        }

        // if the user presses the `q` key, quit the loop
        if ((cv::waitKey(wait_time) & 0xff) == 'q') {
            break;
        }
    }

Line 92, cv::imshow(window_name, cd_frame); simply indicates OpenCV to visualize our frame in the window we created before. This is done only if the frame is not empty.

The other lines added in the loop are used to pace the visualization, so that it is synchronized with the frame generator. To do so, we use the wait_time variable, defined on line 74, which contains the length in milliseconds between two frames. The cv::waitKey(wait_time) function will stop the processing for the defined number of milliseconds, and listen to any key pressed by the user.

95
96
97
98
        // if the user presses the `q` key, quit the loop
        if ((cv::waitKey(wait_time) & 0xff) == 'q') {
            break;
        }

Here we listen to the user for the q key, and use this command to exit the loop, effectively terminating the software. This finally allows us to stop our sample even with a live camera, something that was not possible in the previous versions of this sample.

To summarize, in this section, we learned how to add another callback to the Camera and use the Metavision::CDFrameGenerator to create frames out of the events and display them.

Next steps

In this page, we introduced a minimal example of how to use the Metavision SDK to open a camera, compute some simple statistics on the received events, and visualize the events.

Note that the samples presented here were designed for simplicity. This comes at the cost of performance. For example, performing complex operations directly in the callback is not advised: the callback blocks the Metavision::Camera instance until the processing is finished, something that could create memory issues. Moreover, there is no guarantee that a callback happens at a regular interval, which might be useful for some algorithms. A better approach could be to store events for later use, passing them from class to class, and setting up threads to perform parallel computation, in a similar approach to Metavision Designer

To simplify coding complex applications, we created the Metavision::Pipeline class that transparently takes care of events flow and multi-threading. For more information, go to the pipeline section.

We also encourage you to discover more complex examples in our samples page.