Intrinsics Calibration¶

This application is installed in <install-prefix>/share/metavision/sdk/calibration/apps/metavision_mono_calibration. It allows us to calibrate the intrinsic parameters of an event-based camera.

The application requires a rigid pattern, whose 3D geometry is known. Currently, the application can detect either a blinking checkerboard or a grid of blinking dots. The following HTML pages are provided to visualize these blinking patterns on a screen, and they allow any device to be used for the calibration (e.g. smartphone, tablet, laptop, etc.):

<install-prefix>/share/metavision/sdk/calibration/apps/metavision_mono_calibration/metavision_calibration_pattern_chessboard.html
<install-prefix>/share/metavision/sdk/calibration/apps/metavision_mono_calibration/metavision_calibration_pattern_dots.html

Note that the dots calibration pattern (metavision_calibration_pattern_dots.html) requires a browser that correctly renders a precise frequency. If it is not the case for a particular device, a message “Unstable frequency” will be displayed.

During the calibration process, the calibration pattern should be captured from different distances and angles to cover the camera’s field of view. For each point of view, we recommend to keep the camera static for few seconds for a better pattern detection. Ideally, you should mount the camera on a tripod and then move the camera together with the tripod or move the screen displaying the blinking pattern. Once the pattern is captured from one viewing angle, the camera should be moved to a new position to capture the pattern from a different viewing angle. The process should be repeated as many times as needed to cover the camera’s field of view.

When 50 calibration patterns are acquired or whenever you request it by pressing “c” key, the application will try to determine the parameters of a given mathematical camera model that best describes the relation between the points in the space and the corresponding triggered events.

Some variants of the calibration application are already implemented and available, and some others might be implemented in future releases.

Expected Output¶

Metavision Intrinsics Calibration application visualizes the events generated by the camera (on the left) and the output (on the right) with the current detected pattern, the total number of successfully detected patterns and their overlay indicating the coverage of the field of view:

How to start¶

You could either use the pre-compiled executable or compile the source code as described in this tutorial.

To start the pre-compiled executable based on the live stream from your camera pointing to the provided HTML blinking checkerboard, you need to give the corresponding number of columns and rows as command line options:

Linux

metavision_mono_calibration --cols 11 --rows 6

Windows

metavision_mono_calibration.exe --cols 11 --rows 6

To start the pre-compiled executable based on recorded data, provide the full path to a RAW file instead of FILE_NAME:

Linux

metavision_mono_calibration -i FILE_NAME

Windows

metavision_mono_calibration.exe -i FILE_NAME

To check for additional options:

Linux

metavision_mono_calibration -h

Windows

metavision_mono_calibration.exe -h

Code Overview¶

Pipeline¶

The application implements the following pipeline:

Trail Stage¶

The Metavision::TrailFilterAlgorithm is used as a first stage to filter the noise and reduce the number of events to process. The algorithm is initialized with a wide time window which basically results in keeping the events corresponding to polarity changes or new edges.

The filtered events are sent to the next stages.

Pattern Detector Stage¶

This stage is in charge of detecting a known pattern from input CD events. When the pattern is detected, this stage produces a Metavision::CalibrationDetectionResult which gathers the 2D detections of the pattern’s keypoints and a frame of the detected pattern for visualization.

Two pattern detectors are currently implemented:

A blinking checkerboard that uses the Metavision::BlinkingFrameGeneratorAlgorithm to reconstruct a frame from a blinking pattern, and then applies standard computer vision algorithms to detect it.
A blinking dots detector that uses the Metavision::BlinkingDotsGridDetectorAlgorithm to detect a grid of dots blinking at different frequencies.

Note

A command line option allows to select the pattern detector to use. We see here that we can easily switch the implementation of a component without modifying the whole pipeline if the same input(s) and output(s) are used.

These pattern detectors are asynchronous, as they only produce a result when the pattern is detected. The processing is done in the consuming callback of the stage:

set_consuming_callback([this](const boost::any &data) {
    try {
        auto buffer = boost::any_cast<EventBufferPtr>(data);
        if (buffer->empty())
            return;
        successful_cb_ = false;
        algo_->process_events(buffer->cbegin(), buffer->cend());
        if (!successful_cb_)
            produce(std::make_pair(buffer->crbegin()->t, CalibResultsPtr())); // Temporal marker
    } catch (boost::bad_any_cast &c) { MV_LOG_ERROR() << c.what(); }
});

The result is produced in the output callback passed to the algorithm:

algo_->set_output_callback(
    [this](Metavision::timestamp ts, Metavision::CalibrationDetectionResult &pattern_detection) {
        auto output_ptr = calib_pool_.acquire();
        std::swap(*output_ptr, pattern_detection);
        successful_cb_ = true;
        produce(std::make_pair(ts, output_ptr));
    });

Note

The fact that the detection result is passed to the output callback via a non constant reference allows us to swap it to avoid useless copies. This way the pattern detector can attempt new detections, while the current one is sent without any copy to the next stages.

Note

In the code snippets above, the successful_cb_ flag is used to detect when the algorithm doesn’t detect a pattern. In that case we send an empty detection result, that acts as a temporal marker, to the next stages to ease the synchronization later on.

Calibration Stage¶

This stage is in charge of collecting all the detection results that will be used to calibrate the camera. The detection results are then forwarded to the next stages without modification. Some detection results might be filtered out if too close to the previous ones. In that case, we send an empty detection to the next stages to ease the synchronization and act as a temporal marker:

set_consuming_callback([this](const boost::any &data) {
    try {
        auto ts_calib_results     = boost::any_cast<CalibResultsData>(data);
        auto &input_ts            = ts_calib_results.first;
        auto &input_calib_results = ts_calib_results.second;
        if (input_calib_results) {
            if (calib_algo_->add_2d_detection(input_ts, input_calib_results->keypoints_)) {
                produce(data);
                return;
            }
        }
        produce(std::make_pair(input_ts, CalibResultsPtr())); // Temporal marker
    } catch (boost::bad_any_cast &c) { MV_LOG_ERROR() << c.what(); }
});

Note

Several mathematical camera models can be considered to calibrate a camera. Currently, only the pinhole camera model with radial and tangential distortion is implemented (i.e. in the Metavision::PinholeMonoCalibrator which is based on OpenCV). However, like what is done for the pattern detector, the application can easily be adapted to provide new calibration stages implementing different camera models.

When either the record is over or the user presses the key c, the pipeline ends and the Metavision::PinholeMonoCalibrator::perform_calibration() method is called to perform the calibration. If the calibration succeeds, the result is written to the output directory, which was passed as argument to the application.

Pattern Frame Generator Stage¶

This stage uses the Metavision::CalibrationDetectionFrameGenerator to generate a frame showing the pattern detections. A new frame is generated for each input Metavision::CalibrationDetectionResult. The Metavision::CalibrationDetectionFrameGenerator adds an overlay on top of the image given in the Metavision::CalibrationDetectionResult which represents all the previous pattern detections.

As seen in the previous stage, an empty detection result might be received. It corresponds to a temporal marker that needs to be forwarded, a empty frame is thus sent to the next stages:

set_consuming_callback([this](const boost::any &data) {
    try {
        auto ts_calib_results = boost::any_cast<CalibResultsData>(data);
        auto output_frame_ptr = frame_pool_.acquire();

        const auto &input_ts            = ts_calib_results.first;
        const auto &input_calib_results = ts_calib_results.second;
        if (!input_calib_results) {
            produce(std::make_pair(input_ts, FramePtr())); // Temporal marker
            return;
        }
        display_algo_->generate_bgr_img(*output_frame_ptr, *input_calib_results);
        produce(std::make_pair(input_ts, output_frame_ptr));
    } catch (boost::bad_any_cast &c) { MV_LOG_ERROR() << c.what(); }
});

Frame Generation Stage¶

This stage, implemented in the Metavision::FrameGenerationStage class, uses the Metavision::FrameGenerationAlgorithm to generate a frame from the events. The events are directly drawn in the frame upon reception and the frame is produced (that is, sent to the next stages) at a fixed frequency in the camera’s clock.

Note

This approach is more efficient than the one implemented in Metavision::EventsFrameGeneratorAlgorithm where the events are buffered before being drawn. However, this later approach eases the synchronization.

Frame Composition Stage¶

This stage uses the Metavision::FrameCompositionStage class to display, side by side, the two frames generated up to now: the events frame, produced by the frame generator stage, and the detections frame, produced by the pattern frame generator stage.

The previous stages are connected using the Metavision::FrameCompositionStage::add_previous_frame_stage() method:

auto &frame_composer_stage = p.add_stage(std::make_unique<Metavision::FrameCompositionStage>(display_fps));
frame_composer_stage.add_previous_frame_stage(high_freq_score_stage, width + 10, 0, header_score_width,
                                              header_score_height);
frame_composer_stage.add_previous_frame_stage(events_frame_stage, 0, header_score_height + 10, width, height);
frame_composer_stage.add_previous_frame_stage(blinking_frame_generator_stage, width + 10, header_score_height + 10,
                                              width, height);

The composed frame is produced at a fixed frequency in the camera’s clock in contrast to the input frames that might arrive at different and variable frequencies. Variable frequencies happen because of asynchronous algorithms that might produce 0, 1 or N output(s) for each input.

Temporal markers are used to ease the synchronization that is done internally in the Metavision::FrameCompositionStage.

Display Stage¶

The frame produced by the image composer stage is displayed in this stage:

While the events frame helps you to know what is happening in real time, the detections frame indicates which areas of the image plane have been well covered so far. Ensuring the whole image plane has been covered helps to get good calibration results.