Active Marker 3D Tracking using C++

Overview

The Computer Vision API can be used to detect and track active markers in 3D.

The sample metavision_active_marker_3d_tracking shows how to implement a pipeline for detecting and tracking an active marker in 3D.

The source code of this sample can be found in <install-prefix>/share/metavision/sdk/cv3d/cpp_samples/metavision_active_marker_3d_tracking when installing Metavision SDK from installer or packages. For other deployment methods, check the page Path of Samples.

Expected Output

The sample displays, in 3D, the trajectory of the active marker with respect to the camera (or the other way around).



How to start

First, compile the sample as described in this tutorial.

To run the sample, you need to provide:

  • a JSON file containing the description of the active marker you want to track

  • a JSON file containing the intrinsics calibration of the event-based camera

  • a 3D scene file representing the active marker to be tracked and the scene. The 3D scene must be provided in the OGRE’s format (i.e. .scene file)

  • also, if you want to start the sample based on the live stream from your camera, you will need to provide a camera setting file that tunes the biases to values wisely chosen for this specific application

In general, there is not a defined unit length for 3D files. The user can apply the unit of his choice as long as a consistency between the JSON file and the mesh file is kept.

For the provided 3D files, we would consider “millimeter” as dimension unit, due to the size of the LED board used.

The easiest way to try this sample is to launch it on the active_marker.raw file that we provide in our Sample Recordings. Download the file active_marker.zip which contains both the RAW and the active marker description JSON file. Assuming you extracted the archive next to your executable, you can launch the sample with the following command:

Linux

./metavision_active_marker_3d_tracking -i active_marker_dataset/active_marker.raw -a active_marker_dataset/rectangular_active_marker.json -c active_marker_dataset/calibration.json -s active_marker_dataset/handle_scene/handle_scene.scene

Windows

metavision_active_marker_3d_tracking.exe -i active_marker_dataset/active_marker.raw -a active_marker_dataset/rectangular_active_marker.json -c active_marker_dataset/calibration.json -s active_marker_dataset/handle_scene/handle_scene.scene

Note

The RAW recording file in the dataset captures a handheld active marker moving in front of the camera. This file is best suited for use with the “handle scene”, which demonstrates active markers in a VR context.

The “industrial scene” showcases active markers in an industrial context, where an active marker is used to track a container attached to the end of a crane.

If you want to start the sample based on the live stream from your camera, you will need to provide a camera settings file with bias values wisely chosen for this specific application. To do so, you can use the command line option --input-camera-config (or -j) with a JSON file containing the settings. To create such a JSON file, check the camera settings section.

Here is how to launch the sample with a JSON camera settings file:

Linux

./metavision_active_marker_3d_tracking -j <path_to_a_camera_settings_file> -a active_marker_dataset/rectangular_active_marker.json -c active_marker_dataset/calibration.json -s active_marker_dataset/handle_scene/handle_scene.scene

Windows

metavision_active_marker_3d_tracking.exe -j <path_to_a_camera_settings_file> -a active_marker_dataset/rectangular_active_marker.json -c active_marker_dataset/calibration.json -s active_marker_dataset/handle_scene/handle_scene.scene

To check for additional options:

Linux

./metavision_active_marker_3d_tracking -h

Windows

metavision_active_marker_3d_tracking.exe -h

An important option to consider is --update-mode, which sets the point of vue of the 3d viewer:

  • Object: The camera remains stationary, the object is moving from the camera point of vue.

  • VR: The camera remains stationary, the object is moving from the user point of vue.

  • Camera: The camera moves within the scene, while the object remains stationary.

Code Overview

The Metavision Active Marker 3D Tracking sample implements the following data flow:

Metavision Active Marker 3D tracking Data Flow

  • The first step is to start the camera or open a record file. In case of a live stream, the camera settings file is loaded to properly configure the camera for this application. Note that the bias range check is disabled to allow configuring the camera with values specific to this application:

Metavision::Camera camera;
std::array<uint16_t, 2> geometry;
if (file_in_pos) {
    file_in_pos->read((char *)geometry.data(), sizeof(geometry));
    std::cerr << "Reading Camera geometry: " << geometry[0] << "x" << geometry[1] << "\n";
} else {
    if (opt_config->event_file_path.empty()) {
        Metavision::DeviceConfig device_config;
        device_config.enable_biases_range_check_bypass(true);
        camera = Metavision::Camera::from_first_available(device_config);
        camera.load(opt_config->cam_config_path);
    } else {
        const auto cam_config =
            Metavision::FileConfigHints().real_time_playback(opt_config->realtime_playback_speed);
        camera = Metavision::Camera::from_file(opt_config->event_file_path, cam_config);
    }
    geometry = {static_cast<std::uint16_t>(camera.geometry().get_width()),
                static_cast<std::uint16_t>(camera.geometry().get_height())};
}

  • Then, the description of the Active Marker to be used is loaded from the given JSON file. Here, both the IDs and the 3D positions of the LEDs are loaded:

const auto active_marker = detail::load_active_marker(opt_config->am_json_path);

  • Then, the intrinsics calibration of the event-based camera is loaded:

const auto camera_geometry = Metavision::load_camera_geometry<float>(opt_config->calib_json_path);

  • The 3D viewer and the algorithms are instantiated and configured:

Metavision::Viewer3d viewer(opt_config->viewer_params);
Metavision::ModulatedLightDetectorAlgorithm modulated_light_detector(opt_config->detector_params);
Metavision::ActiveMarkerTrackerAlgorithm tracker(opt_config->tracker_params, detail::to_set(active_marker));
Metavision::ActiveMarkerPoseEstimatorAlgorithm pose_estimator(opt_config->pose_estimator_params, *camera_geometry,
                                                              active_marker);

  • A time callback is added to the camera device. This is very useful as it allows the pose estimator algorithm, and under the hood, the tracking algorithm, to be regularly notified that some amount of time has elapsed since the last received events. This is very important as it allows the tracking algorithm to execute its internal processes even when no events are received:

auto decoder = camera.get_device().get_facility<Metavision::I_EventsStreamDecoder>();
decoder->add_time_callback([&](Metavision::timestamp t) { tracker.notify_elapsed_time(t); });

  • The processing callback is set to the camera which executes the algorithms:

std::vector<Metavision::EventSourceId> source_id_events;
std::vector<Metavision::EventActiveTrack> track_events;
camera.cd().add_callback([&](const auto begin, const auto end) {
    source_id_events.clear();
    track_events.clear();

    modulated_light_detector.process_events(begin, end, std::back_inserter(source_id_events));
    tracker.process_events(source_id_events.cbegin(), source_id_events.cend(), std::back_inserter(track_events));
    pose_estimator.process_events(track_events.cbegin(), track_events.cend());
});

  • A callback is set to the pose estimator to update the 3D viewer when a new pose is available:

pose_estimator.set_pose_update_callback(
    [&](Metavision::timestamp t, const Metavision::ActiveMarkerPoseEstimatorAlgorithm::PoseUpdate &T_w_m) {
        viewer.apply_pose_update(T_w_m);
    });

  • Finally the camera and the 3D viewer are started:

camera.start();
viewer.run();
camera.stop();

Algorithms Overview

Active Marker Pose Estimator

The Active Marker 3D Tracking sample uses the same algorithms as the ones used in the Active Marker 2D Tracking Sample. The difference here is that an additional algorithm brick, the Metavision::ActiveMarkerPoseEstimatorAlgorithm which is built on top of the Metavision::ActiveMarkerTrackerAlgorithm, computes the pose of the Active Marker with respect to the event-based camera.

To do so, the Metavision::ActiveMarkerPoseEstimatorAlgorithm consumes Metavision::EventSourceId events which are used to feed the Metavision::ActiveMarkerTrackerAlgorithm internally. The Metavision::EventActiveTrack events produced are then associated with the 3D positions of the LEDs of the Active Marker (set during the initialization of the algorithm) to compute the marker’s pose with respect to the event-based camera. The pose is initialized by solving the well-known PnP problem and subsequently updated by estimating its velocity.

The pose can be computed in several ways:

  • every N events

  • every N microseconds in the event-based camera’s clock

  • every N microseconds in the system’s clock

The first option allows adapting the pose computation rate to the speed of the camera while the last two allow updating the pose at a fixed frequency which can be useful in cases with real-time constraints. The pose estimation mode is set once when the algorithm is instantiated and then, independently from the chosen mode, a callback is called every time a new pose update is available.

Configuring the camera

Please refer to the Active Marker 2D Tracking Sample for a guide on how to configure the camera.