SDK CV3D Algorithms

class ActiveMarkerPoseEstimatorAlgorithm

A class that estimates the 3D pose of an active marker with respect to the camera.

An active marker is a rigid object composed of several blinking LEDs. Knowing the 3D location of each LED in the coordinates system made by the active marker itself, its 3D pose can be recovered by tracking the 2D location of the LEDs in the sensor plane. The 3D pose can be computed either in the camera’s clock (every N events or every N us) or in the system’s clock (at a given frequency).

Public Types

enum class EstimatorType

Clock in which the 3D pose is estimated.

Values:

enumerator CameraClock

enumerator SystemClock

Public Functions

ActiveMarkerPoseEstimatorAlgorithm(const Params &params, const CameraGeometryBase<float> &camera_geometry, const std::unordered_map<std::uint32_t, cv::Point3f> &active_marker_leds)

Constructor.

Parameters

params – The parameters for configuring the estimator
camera_geometry – The camera geometry of the event-based camera
active_marker_leds – The map of LED IDs to their 3D locations in the active marker’s coordinate system

~ActiveMarkerPoseEstimatorAlgorithm(): Destructor.

void set_pose_update_callback(PoseUpdateCallback cb)

Sets the callback function for receiving pose updates.

Note

If the pose estimation fails (e.g. not enough visible LEDs), the callback is still called but with an empty pose update. This is a way to get notified when the tracking gets lost.

Parameters: cb – The callback function to be called

template<typename InputIt> void process_events(InputIt begin, InputIt end)

Processes a range of EventSourceId events for updating the pose estimation.

Template Parameters

InputIt – Input event iterator type, works for iterators over containers of EventSourceId

Parameters

begin – Iterator pointing to the first event in the stream
end – Iterator pointing to the past-the-end element in the stream

struct Params

Parameters for configuring the ActiveMarkerPoseEstimator.

In the camera’s clock mode, if delta_n_events and delta_ts parameters are valid, then delta_n_events will be used

Public Members

EstimatorType type = EstimatorType::CameraClock : Clock in which the 3D pose is estimated.

size_t delta_n_events = 5000: Number of events between two pose estimates in the Camera’s clock.

timestamp delta_ts = 0: Time interval (in us) between two pose estimates in the Camera’s clock.

float estimation_rate_hz = 0.f: Pose estimation rate (in Hz) in the System’s clock.

std::uint8_t pose_buffer_length = 6: Size of the rolling window used for smoothing pose estimates.

std::uint16_t track_max_age = 1000: Maximum age of an event (in us) to be used in the pose estimation process

Public Static Functions

static Params make_n_events_camera_clock(size_t delta_n_events, std::uint8_t pose_buffer_length = 6, std::uint16_t track_max_age = 1000)

Creates parameters for a camera clock-based estimator using the number of events as the trigger for pose updates.

Parameters

delta_n_events – The number of events between pose updates
pose_buffer_length – Size of the rolling window used for smoothing pose estimates
track_max_age – Maximum age of an event (in us) to be used in the pose estimation process

Returns

The parameters

static Params make_n_us_camera_clock(timestamp delta_ts, std::uint8_t pose_buffer_length = 6, std::uint16_t track_max_age = 1000)

Creates parameters for a camera clock-based estimator using a time interval as the trigger for pose updates.

Parameters

delta_ts – The time interval (in us) between pose updates
pose_buffer_length – Size of the rolling window used for smoothing pose estimates
track_max_age – Maximum age of an event (in us) to be used in the pose estimation process

Returns

The parameters

static Params make_system_clock(float estimation_rate, std::uint8_t pose_buffer_length = 6, std::uint16_t track_max_age = 1000)

Creates parameters for a system clock-based pose estimator.

Parameters

estimation_rate – The rate (in Hz) at which pose updates are estimated
pose_buffer_length – Size of the rolling window used for smoothing pose estimates
track_max_age – Maximum age of an event (in us) to be used in the pose estimation process

Returns

The parameters

struct PoseSerializer

Structure to serialize the pose represented by a 4x4 matrix.

Public Functions

PoseSerializer() = default: Default constructor.

inline PoseSerializer(const PoseUpdate &pos)

Constructor.

Parameters: pos – The pose update

Public Members

char has_pos = 0x00: Whether the pose has a value.

Eigen::Matrix4f m = {}: The 4x4 matrix representing the pose

Friends

inline friend std::ostream &operator<<(std::ostream &out, const PoseSerializer &p): Serializes the pose to an output stream.

class Edgelet2dDetectionAlgorithm

Algorithm used to detect 2D edgelets in a time surface.

Public Functions

inline Edgelet2dDetectionAlgorithm(timestamp threshold = 0)

Constructor.

See also

is_fast_edge for more details

Parameters: threshold – Detection tolerance threshold

~Edgelet2dDetectionAlgorithm() = default: Destructor.

template<typename InputIt, typename OutputIt> OutputIt process(const MostRecentTimestampBuffer &ts, InputIt begin, InputIt end, OutputIt d_begin)

Tries to detect 2D edgelets in the time surface at locations given by the input events.

Template Parameters

InputIt – Read-Only input event iterator type
OutputIt – Read-Write output EventEdgelet2d event iterator type

Parameters

ts – Time surface in which 2D edgelets are looked for
begin – First iterator to the buffer of events whose locations will be looked at to detect edgelets
end – Last iterator to the buffer of events whose locations will be looked at to detect edgelets
d_begin – Output iterator of 2D edgelets buffer

Returns

Iterator pointing to the past-the-end event added in the output 2D edgelets buffer

class Edgelet2dTrackingAlgorithm

Algorithm used to track 2D edgelets in a time surface.

Points are sampled along the edgelet’s direction (a.k.a support points) and matches are looked for on the two sides of the edgelet along its normal. A match is found when a timestamp more recent than a target timestamp if found in the time surface. A new line is then fitted from those matches and both a new direction and normal are computed.

Warning

Because of the aperture problem, the tracking can drift very quickly. As a result, this algorithm should only be used if there are methods that constrain the tracking or to track edgelets that are orthogonal to the camera’s motion.

Public Functions

Edgelet2dTrackingAlgorithm(const Parameters &params = Parameters())

Constructor.

Parameters: params – Parameters used by the tracking algorithm

~Edgelet2dTrackingAlgorithm() = default: Destructor.

template<typename Edgelet2dIt, typename StatusIt, typename OutputEdgelet2dIt> OutputEdgelet2dIt process(const MostRecentTimestampBuffer &time_surface, timestamp target, Edgelet2dIt edgelet_begin, Edgelet2dIt edgelet_end, OutputEdgelet2dIt d_begin, StatusIt status_begin)

Tracks the input 2D edgelets in the input time surface.

For each input 2D edgelet a status is updated to indicate whether the corresponding edgelet has been tracked or not. When an edgelet is successfully tracked, an updated 2D edgelet (i.e. corresponding to the matched edgelet in the time surface) is outputted. This means that a user needs to either, pass a std::back_insert_iterator to these two buffers, or, pre-allocate them with the same size as the input 2D edgelets buffer’s one.

Warning

The output matched 2D edgelets buffer needs to be resized to only contain the matched edgelets, or a std::back_insert_iterator needs to be passed instead.

Edgelet2dTrackingAlgorithm algo;
MostRecentTimestampBuffer time_surface(height, width, n_channels);
std::vector<EventEdgelet2d> detected_edgelets;
// Detect edgelets
...
std::vector<EventEdgelet2d> tracked_edgelets(detected_edgelets.size());
std::vector<bool> statuses(detected_edgelets.size());

auto tracked_edglet_end = algo.process(time_surface, target_ts, detected_edgelets.cbegin(),
                                       detected_edgelets.cend(), tracked_edgelets.begin(), statuses.begin());
tracked_edgelets.resize(std::distance(tracked_edgelets.end(), tracked_edglet_end));

or

Edgelet2dTrackingAlgorithm algo;
MostRecentTimestampBuffer time_surface(height, width, n_channels);
std::vector<EventEdgelet2d> detected_edgelets;
// Detect edgelets
...
std::vector<EventEdgelet2d> tracked_edgelets;
std::vector<bool> statuses);

auto tracked_edglet_end = algo.process(time_surface, target_ts, detected_edgelets.cbegin(),
                                       detected_edgelets.cend(),
                                       std::back_insert_iterator(tracked_edgelets.begin()),
                                       std::back_insert_iterator(statuses.begin()));

Template Parameters

Edgelet2dIt – Iterator type of the input 2D edgelets
StatusIt – Iterator type of the input statuses
OutputEdgelet2dIt – Iterator type of the output 2D edgelets

Parameters

time_surface – The time surface in which 2D edgelets are looked for
target – Target timestamp used for matching. A timestamp ts in the time surface will match a support point if ts > (target - threshold_)
edgelet_begin – First iterator to the buffer of 2D edgelets that will be looked for
edgelet_end – Last iterator to the buffer of 2D edgelets that will be looked for
d_begin – First iterator to the matched 2D edgelets buffer
status_begin – First iterator to the tracking statuses buffer

Returns

The last iterator to the matched 2D edgelets

const Parameters &get_parameters() const: Returns the algorithm’s parameters.

struct Parameters

Parameters used by the tracking algorithm.

Public Members

unsigned int search_radius_ = 3: Radius in which matches are looked for each edgelet’s side.

unsigned int n_support_points_ = 3: Number of points sampled along the edgelet’s direction.

float support_points_distance_ = 2: Distance in pixels between the sampled points.

timestamp threshold_ = 3000: Time tolerance used in the tracking.

unsigned int median_outlier_threshold_ = 1: Distance to the median position of the support points’ matches above which a match is considered as outlier.

class Model3dDetectionAlgorithm

Algorithm that detects a known 3D model by detecting its edges in an events stream.

Support points are sampled along the 3D model’s visible edges and tracked in a time surface in which the events stream has been accumulated. Matches are looked for in the time surface by looking for timestamps on slopes that have been generated by moving edges having the same orientations as the 3D model’s ones. An edge is considered matched when a line can be fitted from its matches. When enough edges are matched, the 3D model’s pose is estimated by minimizing the orthogonal distance between the matches and their corresponding reprojected edge.

Public Functions

Model3dDetectionAlgorithm(const CameraGeometry32f &cam_geometry, const Model3d &model, MostRecentTimestampBuffer &time_surface, const Parameters &params = Parameters())

Constructor.

Parameters

cam_geometry – Camera geometry instance allowing mapping coordinates from camera to image (and vice versa)
model – 3D model to detect
time_surface – Time surface instance in which the events stream is accumulated
params – Algorithm’s parameters

void set_init_pose(const Eigen::Matrix4f &T_c_w)

Sets the camera’s pose from which the algorithm will try to detect the 3D model.

Parameters: T_c_w – Camera’s initialization pose

template<typename InputIt> bool process_events(InputIt it_begin, InputIt it_end, Eigen::Matrix4f &T_c_w, std::set<size_t> *visible_edges = nullptr, std::set<size_t> *detected_edges = nullptr)

Tries to detect the 3D model from the input events buffer.

Note

The estimated pose might be not very accurate but accurate enough to initiate a tracking phase

Template Parameters

InputIt – Read-Only input event iterator type.

Parameters

it_begin – [in] Iterator to the first input event
it_end – [in] Iterator to the past-the-end event
T_c_w – [out] Camera’s pose if the detection has succeeded
visible_edges – [out] If filled, contains the model’s edges visible from the initialization pose
detected_edges – [out] If filled, contains the model’s successfully detected edges

Returns

True if the detection has succeeded, false otherwise

struct Parameters

Parameters used by the model 3d detection algorithm.

Public Members

std::uint32_t support_point_step_ = 10: Distance, in pixels in the distorted image, between two support points

std::uint32_t search_radius_ = 10: Radius in which matches are looked for for each support point.

std::uint32_t flow_radius_ = 3: Radius used to estimate the normal flow which gives the edge’s orientation

std::uint32_t n_fitting_pts_ = 3: Minimum required number of matches for line fitting.

float variance_threshold_ = 2e-5f: Variance of the support points around the fitted line below which an edge is considered matched

float fitted_edges_ratio_ = 0.5f: Matched edges to visible edges ratio above which a pose estimation is attempted

class Model3dTrackingAlgorithm

Algorithm that estimates the 6 DOF pose of a 3D model by tracking its edges in an events stream.

Support points are sampled along the 3D model’s visible edges and tracked in a time surface in which the events stream has been accumulated. Matches are looked for in the time surface within a fixed radius and a given accumulation time. A weight is then attributed to every support point according to the timestamp of the event they matched with. Finally, the pose is estimated using a weighted least squares to minimize the orthogonal distance between the matches and their corresponding reprojected edge.

The accumulation time used for matching can vary depending on how the algorithm is called. Indeed, the algorithm computes the accumulation time by maintaining a sliding buffer of the last N pose computation timestamps. As a result, the accumulation time will be fixed when the algorithm is called every N us and varying when called every N events. The latter is more interesting because, in that case, the accumulation will adapt to the motion of the camera. In case of fast motion, the tracking will restrict the matching to very recent events making it more robust to noise. Whereas in case of slow motion, the tracking will allow matching older events.

Public Functions

Model3dTrackingAlgorithm(const CameraGeometry32f &cam_geometry, const Model3d &model, MostRecentTimestampBuffer &time_surface, const Parameters &params = Parameters())

Constructor.

Parameters

cam_geometry – Camera geometry instance allowing mapping coordinates from camera to image (and vice versa)
model – 3D model to track
time_surface – Time surface instance in which the events stream is accumulated
params – Algorithm’s parameters

void set_previous_camera_pose(timestamp ts, const Eigen::Matrix4f &T_c_w)

Initializes the tracking by setting a camera’s prior pose.

Parameters

ts – Timestamp at which the pose was estimated
T_c_w – Camera’s prior pose

template<typename InputIt> bool process_events(InputIt it_begin, InputIt it_end, Eigen::Matrix4f &T_c_w)

Tries to track the 3D model from the input events buffer.

Template Parameters

InputIt – Read-Only input event iterator type.

Parameters

it_begin – [in] Iterator to the first input event
it_end – [in] Iterator to the past-the-end event
T_c_w – [out] Camera’s pose if the tracking has succeeded

Returns

True if the tracking has succeeded, false otherwise

struct Parameters

Parameters used by the model 3d tracker algorithm.

Public Members

std::uint32_t search_radius_ = 3: Radius (in pix) in which matches are looked for for each support point.

std::uint32_t support_point_step_ = 10: Distance, in pixels in the distorted image, between two support points

std::uint32_t n_last_poses_ = 5: Number of past poses to consider to compute the accumulation time.

timestamp default_acc_time_us_ = 3000: Default accumulation time used when the tracking is starting (i.e. the N last poses have not been estimated yet)

float oldest_weight_ = 0.1f: Weight attributed to the oldest matches.

float most_recent_weight_ = 1.f: Weight attributed to the more recent matches.

std::uint32_t nb_directional_axes_ = 8: Number of pre-computed axes to consider to quantize the normal of an edge.

class PyramidalStereoBlockMatchingAlgorithm

Class to compute stereo disparities using a pyramidal block matching algorithm.

Public Functions

PyramidalStereoBlockMatchingAlgorithm(const Eigen::Vector2i &size_master, const Eigen::Vector2i &size_slave, int plevel_max = 3, int patch_radius = 3, float min_contrast_to_second_best = 1.2f, float max_ssd = std::numeric_limits<float>::max(), bool enable_low_confidence_matches = false)

Constructor.

Parameters

size_master – Size of the master image
size_slave – Size of the slave image
plevel_max – Maximum pyramid level
patch_radius – Radius of the square patch around each pixel of the master image to be matched in the slave image.
min_contrast_to_second_best – Minimum contrast ratio between the best and second best matches to consider the best match as valid.
max_ssd – Maximum sum of squared differences to consider the best match as valid.
enable_low_confidence_matches – Flag to enable the inclusion of low-confidence matches in the results, such as matches where subpixel refinement failed or for which the disparity search range was truncated. False by default.

~PyramidalStereoBlockMatchingAlgorithm() = default: Destructor.

void set_disparity_range(const StereoCameraGeometry<float>::RectifiedStereoGeometry &stereo_rectified_geometry, int min_disp, int max_disp): Sets the disparity range.

void set_depth_range(const StereoCameraGeometry<float>::RectifiedStereoGeometry &stereo_rectified_geometry, float min_depth, float max_depth): Sets the disparity range based on the stereo geometry and the specified depth range.

void get_disparity_range(int &min_disp, int &max_disp) const: Gets the disparity range.

void compute_disparities(const cv::Mat_<uchar> &rectified_img_master, const cv::Mat_<uchar> &rectified_img_slave, cv::Mat_<float> &disparity_map_master, const cv::Mat_<uchar> *rectified_mask_master = nullptr)

Computes the disparity map between two provided rectified images.

Parameters

rectified_img_master – [in] The rectified master image
rectified_img_slave – [in] The rectified slave image
disparity_map_master – [out] The output disparity map
rectified_mask_master – [in] Optional mask of pixels to be matched from the rectified master image

class SimpleStereoBlockMatchingAlgorithm

Class to compute stereo disparities using a single-level block matching algorithm.

Public Functions

SimpleStereoBlockMatchingAlgorithm(int patch_radius = 3, float min_contrast_to_second_best = 1.2f, float max_ssd = std::numeric_limits<float>::max(), bool enable_low_confidence_matches = false)

Constructor.

Parameters

patch_radius – Radius of the square patch around each pixel of the master image to be matched in the slave image.
min_contrast_to_second_best – Minimum contrast ratio between the best and second best matches to consider the best match as valid.
max_ssd – Maximum sum of squared differences to consider the best match as valid.
enable_low_confidence_matches – Flag to enable the inclusion of low-confidence matches in the results, such as matches where subpixel refinement failed or for which the disparity search range was truncated. False by default.

~SimpleStereoBlockMatchingAlgorithm() = default: Destructor.

void set_disparity_range(const StereoCameraGeometry<float>::RectifiedStereoGeometry &stereo_rectified_geometry, int min_disp, int max_disp): Sets the disparity range.

void set_depth_range(const StereoCameraGeometry<float>::RectifiedStereoGeometry &stereo_rectified_geometry, float min_depth, float max_depth): Sets the disparity range based on the stereo geometry and the specified depth range.

void get_disparity_range(int &min_disp, int &max_disp) const

Gets the disparity range.

Parameters

min_disp – [out] Minimum disparity
max_disp – [out] Maximum disparity

void compute_disparities(const cv::Mat_<uchar> &rectified_img_master, const cv::Mat_<uchar> &rectified_img_slave, cv::Mat_<float> &disparity_map_master, const cv::Mat_<uchar> *rectified_mask_master = nullptr, bool use_disparity_guesses = false)

Computes the disparity map between two provided rectified images.

Note

Invalid disparities are set to / recognized as std::numeric_limits<float>::quiet_NaN().

Parameters

rectified_img_master – [in] The rectified master image
rectified_img_slave – [in] The rectified slave image
disparity_map_master – [inout] The output disparity map
rectified_mask_master – [in] Optional mask of pixels to be matched from the rectified master image
use_disparity_guesses – [in] Whether to use the input disparity map as initial guesses for the disparities to be computed