SDK Core ML Corner Detection API

functions to compute homography reprojection error from Manderscheid, J., Sironi, A., Bourdis, N., Migliore, D., & Lepetit, V. Speed invariant time surface for learning to detect corner points with event-based cameras. CVPR 2019.

class metavision_core_ml.corner_detection.compute_homography_reprojection_error.ComputeKPI(npy_folder_path)

Computes reprojection error for planar scenes (eg: Atis dataset)

Parameters

  • npy_folder_path – npy_folder_path is the path of the folder containing a .npy file for each sequence to

  • type (evaluate. Each file is a list of corners of) – {‘names’: [‘x’, ‘y’, ‘id’, ‘t’], ‘formats’: [‘<u2’, ‘<u2’, ‘<i4’, ‘<i8’], ‘offsets’: [0, 2, 4, 8], ‘itemsize’: 16}

  • above. (.npy files can be easily created using the function convert_csv_to_npy) –

metavision_core_ml.corner_detection.compute_homography_reprojection_error.convert_csv_to_npy(csv_path, overwrite=False)

Converts csv files to npy enable use of EventNpyReader class :param csv_path: path of csv :param overwrite: if csv exists overwrite or not

metavision_core_ml.corner_detection.compute_homography_reprojection_error.homography_error(src_pts, dst_pts)

compute reprojection error between two set of corresponding points by estimating an homography between them :param src_pts: numpy array of 2d corners coordinates in source scene (Nx2) :param dst_pts: numpy array of 2d corners coordinates corresponding supposedly to src_pts in destination scene (Nx2)

Returns

The number of points matched, the reprojection error [0, -1] if no homography was found

metavision_core_ml.corner_detection.compute_homography_reprojection_error.project_coordinates(coordinates, homography)

Using the homography, projects coordinates :param coordinates: 2d numpy array of shape Nx2 :param homography: 3x3 homography matrix

Returns

Projected and normalized coordinates

class to create tracks from corners based on distance in space and time

Image and Corner stream data loader

class metavision_core_ml.corner_detection.corner_video_stream_dataset.CornerVideoDatasetIterator(metadata, height, width, rgb, number_of_heatmaps=10, batch_times=1)

Dataset Iterator streaming images, timestamps and corners

Parameters

  • metadata (object) – path to picture or video

  • height (int) – height of input images / video clip

  • width (int) – width of input images / video clip

  • rgb (bool) – stream rgb videos

  • number_of_heatmaps (int) – The number of heatmaps containing corner locations

  • batch_times (int) – number of timesteps of training sequences

metavision_core_ml.corner_detection.corner_video_stream_dataset.make_corner_video_dataset(path, num_workers, batch_size, height, width, min_length, max_length, number_of_heatmaps=10, rgb=False, seed=None, batch_times=1)

Makes a video/ moving picture dataset.

Parameters

  • path (str) – folder to dataset

  • batch_size (int) – number of video clips / batch

  • height (int) – height

  • width (int) – width

  • min_length (int) – min length of video

  • max_length (int) – max length of video

  • mode (str) – ‘frames’ or ‘delta_t’

  • num_tbins (int) – number of bins in event volume

  • number_of_heatmaps (int) – number of corner heatmaps predicted by the network

  • rgb (bool) – retrieve frames in rgb

  • seed (int) – seed for randomness

  • batch_times (int) – number of time steps in training sequence

metavision_core_ml.corner_detection.corner_video_stream_dataset.pad_collate_fn(data_list)

Here we pad with last image/ timestamp to get a contiguous batch

Here we reuse the GPUSimulator from OpenEB to stream synthetic events.

class metavision_core_ml.corner_detection.data_module.EventToCornerDataModule(hparams)

Simulation gives events + frames + corners

prepare_data_per_node

If True, each LOCAL_RANK=0 will call prepare data. Otherwise only NODE_RANK=0, LOCAL_RANK=0 will prepare data.

allow_zero_length_dataloader_with_multiple_devices

If True, dataloader with zero length within local rank is allowed. Default value is False.

test_dataloader()

An iterable or collection of iterables specifying test samples.

For more information about multiple dataloaders, see this section.

For data processing use the following pattern:

  • download in prepare_data()

  • process and split in setup()

However, the above are only necessary for distributed processing.

Warning

do not assign state in prepare_data

  • test()

  • prepare_data()

  • setup()

Note

Lightning tries to add the correct sampler for distributed and arbitrary hardware. There is no need to set it yourself.

Note

If you don’t need a test dataset and a test_step(), you don’t need to implement this method.

train_dataloader()

An iterable or collection of iterables specifying training samples.

For more information about multiple dataloaders, see this section.

The dataloader you return will not be reloaded unless you set :paramref:`~pytorch_lightning.trainer.trainer.Trainer.reload_dataloaders_every_n_epochs` to a positive integer.

For data processing use the following pattern:

  • download in prepare_data()

  • process and split in setup()

However, the above are only necessary for distributed processing.

Warning

do not assign state in prepare_data

  • fit()

  • prepare_data()

  • setup()

Note

Lightning tries to add the correct sampler for distributed and arbitrary hardware. There is no need to set it yourself.

val_dataloader()

An iterable or collection of iterables specifying validation samples.

For more information about multiple dataloaders, see this section.

The dataloader you return will not be reloaded unless you set :paramref:`~pytorch_lightning.trainer.trainer.Trainer.reload_dataloaders_every_n_epochs` to a positive integer.

It’s recommended that all data downloads and preparation happen in prepare_data().

  • fit()

  • validate()

  • prepare_data()

  • setup()

Note

Lightning tries to add the correct sampler for distributed and arbitrary hardware There is no need to set it yourself.

Note

If you don’t need a validation dataset and a validation_step(), you don’t need to implement this method.

Defines the network’s architecture used in Detecting Stable Keypoints from Events through Image Gradient Prediction and Long-Lived Accurate Keypoints in Event Streams inspired from Fast Image Reconstruction with an Event Camera

class metavision_core_ml.corner_detection.firenet.FireNet(cin=1, cout=1, base=12)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(x, mask=None)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Here we reuse the GPUSimulator from OpenEB to stream synthetic events and corners.

class metavision_core_ml.corner_detection.gpu_corner_esim.GPUEBSimCorners(dataloader, simulator, batch_times, event_volume_depth, randomize_noises, device, number_of_heatmaps, height, width, batch_size)

Simulated Events on GPU returns events, images and corners

Parameters

  • dataloader – video-clips dataloader

  • simulator – gpu-simulator

  • batch_times – number of rounds per batch

  • event_volume_depth – number of timesteps per round

  • device – hardware to run simulation on

classmethod from_params(folder, num_workers, batch_size, batch_times, event_volume_depth, height, width, min_frames_per_video, max_frames_per_video, number_of_heatmaps, randomize_noises=False, device='cuda:0')

Creates the simulator from parameters :param folder: folder of images :param num_workers: number of workers :param batch_size: size of batch :param batch_times: time dimension per batch :param event_volume_depth: number of channels in event volume :param height: height of images :param width: width of images :param min_frames_per_video: minimum number of frames per video :param max_frames_per_video: maximum number of frames per video :param number_of_heatmaps: number of heatmaps of corners locations returned :param randomize_noises: whether or not to randomize the noise of the simulator :param device: location of data

Returns

GPUEBSimCorners class instantiated

randomize_noises(first_times)

Randomizes noise in the simulator consistent with the batches :param first_times: whether or not the video in the batch is new

metavision_core_ml.corner_detection.gpu_corner_esim.collect_target_images(gray_images, timestamps, video_len, target_indices, num_heatmaps)

Collect target frames + timestamps at target indices and rearranges them into T,B,C,H,W tensor

Parameters

  • gray_images (tensor) – H,W,T format (videos are concatenated along 3rd dimension

  • timestamps (tensor) – B,T

  • video_len (tensor) – B lengths

  • target_indices (tensor) – B,M indices

  • num_heatmaps (int) – number of heatmaps

Pytorch Lightning module

class metavision_core_ml.corner_detection.lightning_model.CornerDetectionCallback(data_module, video_result_every_n_epochs=2)

callbacks to our model

on_train_epoch_end(trainer, pl_module)

Called when the train epoch ends.

To access all batch outputs at the end of the epoch, you can cache step outputs as an attribute of the pytorch_lightning.core.LightningModule and access them in this hook:

class MyLightningModule(L.LightningModule):
    def __init__(self):
        super().__init__()
        self.training_step_outputs = []

    def training_step(self):
        loss = ...
        self.training_step_outputs.append(loss)
        return loss


class MyCallback(L.Callback):
    def on_train_epoch_end(self, trainer, pl_module):
        # do something with all training_step outputs, for example:
        epoch_mean = torch.stack(pl_module.training_step_outputs).mean()
        pl_module.log("training_epoch_mean", epoch_mean)
        # free up the memory
        pl_module.training_step_outputs.clear()
class metavision_core_ml.corner_detection.lightning_model.CornerDetectionLightningModel(hparams: argparse.Namespace)

Corner Detection: Train your FireNet model to predict corners as a heatmap

configure_optimizers()

Choose what optimizers and learning-rate schedulers to use in your optimization. Normally you’d need one. But in the case of GANs or similar you might have multiple. Optimization with multiple optimizers only works in the manual optimization mode.

Returns

Any of these 6 options.

  • Single optimizer.

  • List or Tuple of optimizers.

  • Two lists - The first list has multiple optimizers, and the second has multiple LR schedulers (or multiple lr_scheduler_config).

  • Dictionary, with an "optimizer" key, and (optionally) a "lr_scheduler" key whose value is a single LR scheduler or lr_scheduler_config.

  • None - Fit will run without any optimizer.

The lr_scheduler_config is a dictionary which contains the scheduler and its associated configuration. The default configuration is shown below.

lr_scheduler_config = {
    # REQUIRED: The scheduler instance
    "scheduler": lr_scheduler,
    # The unit of the scheduler's step size, could also be 'step'.
    # 'epoch' updates the scheduler on epoch end whereas 'step'
    # updates it after a optimizer update.
    "interval": "epoch",
    # How many epochs/steps should pass between calls to
    # `scheduler.step()`. 1 corresponds to updating the learning
    # rate after every epoch/step.
    "frequency": 1,
    # Metric to to monitor for schedulers like `ReduceLROnPlateau`
    "monitor": "val_loss",
    # If set to `True`, will enforce that the value specified 'monitor'
    # is available when the scheduler is updated, thus stopping
    # training if not found. If set to `False`, it will only produce a warning
    "strict": True,
    # If using the `LearningRateMonitor` callback to monitor the
    # learning rate progress, this keyword can be used to specify
    # a custom logged name
    "name": None,
}

When there are schedulers in which the .step() method is conditioned on a value, such as the torch.optim.lr_scheduler.ReduceLROnPlateau scheduler, Lightning requires that the lr_scheduler_config contains the keyword "monitor" set to the metric name that the scheduler should be conditioned on.

Metrics can be made available to monitor by simply logging it using self.log('metric_to_track', metric_val) in your LightningModule.

Note

Some things to know:

  • Lightning calls .backward() and .step() automatically in case of automatic optimization.

  • If a learning rate scheduler is specified in configure_optimizers() with key "interval" (default “epoch”) in the scheduler configuration, Lightning will call the scheduler’s .step() method automatically in case of automatic optimization.

  • If you use 16-bit precision (precision=16), Lightning will automatically handle the optimizer.

  • If you use torch.optim.LBFGS, Lightning handles the closure function automatically for you.

  • If you use multiple optimizers, you will have to switch to ‘manual optimization’ mode and step them yourself.

  • If you need to control how often the optimizer steps, override the optimizer_step() hook.

classmethod load_from_checkpoint(checkpoint_path)

Primary way of loading a model from a checkpoint. When Lightning saves a checkpoint it stores the arguments passed to __init__ in the checkpoint under "hyper_parameters".

Any arguments specified through **kwargs will override args stored in "hyper_parameters".

Parameters

  • checkpoint_path – Path to checkpoint. This can also be a URL, or file-like object

  • map_location – If your checkpoint saved a GPU model and you now load on CPUs or a different number of GPUs, use this to map to the new setup. The behaviour is the same as in torch.load().

  • hparams_file –

    Optional path to a .yaml or .csv file with hierarchical structure as in this example:

    drop_prob: 0.2
dataloader:
    batch_size: 32

    You most likely won’t need this since Lightning will always save the hyperparameters to the checkpoint. However, if your checkpoint weights don’t have the hyperparameters saved, use this method to pass in a .yaml file with the hparams you’d like to use. These will be converted into a dict and passed into your LightningModule for use.

    If your model’s hparams argument is Namespace and .yaml file has hierarchical structure, you need to refactor your model to treat hparams as dict.

  • strict – Whether to strictly enforce that the keys in checkpoint_path match the keys returned by this module’s state dict. Defaults to True unless LightningModule.strict_loading is set, in which case it defaults to the value of LightningModule.strict_loading.

  • **kwargs – Any extra keyword args needed to init the model. Can also be used to override saved hyperparameter values.

Returns

LightningModule instance with loaded weights and hyperparameters (if available).

Note

load_from_checkpoint is a class method. You should use your LightningModule class to call it instead of the LightningModule instance, or a TypeError will be raised.

Note

To ensure all layers can be loaded from the checkpoint, this function will call configure_model() directly after instantiating the model if this hook is overridden in your LightningModule. However, note that load_from_checkpoint does not support loading sharded checkpoints, and you may run out of memory if the model is too large. In this case, consider loading through the Trainer via .fit(ckpt_path=...).

Example:

# load weights without mapping ...
model = MyLightningModule.load_from_checkpoint('path/to/checkpoint.ckpt')

# or load weights mapping all weights from GPU 1 to GPU 0 ...
map_location = {'cuda:1':'cuda:0'}
model = MyLightningModule.load_from_checkpoint(
    'path/to/checkpoint.ckpt',
    map_location=map_location
)

# or load weights and hyperparameters from separate files.
model = MyLightningModule.load_from_checkpoint(
    'path/to/checkpoint.ckpt',
    hparams_file='/path/to/hparams_file.yaml'
)

# override some of the params with new values
model = MyLightningModule.load_from_checkpoint(
    PATH,
    num_layers=128,
    pretrained_ckpt_path=NEW_PATH,
)

# predict
pretrained_model.eval()
pretrained_model.freeze()
y_hat = pretrained_model(x)
training_step(batch, batch_nb)

Here you compute and return the training loss and some additional metrics for e.g. the progress bar or logger.

Parameters

  • batch – The output of your data iterable, normally a DataLoader.

  • batch_idx – The index of this batch.

  • dataloader_idx – The index of the dataloader that produced this batch. (only if multiple dataloaders used)

Returns

  • Tensor - The loss tensor

  • dict - A dictionary which can include any keys, but must include the key 'loss' in the case of automatic optimization.

  • None - In automatic optimization, this will skip to the next batch (but is not supported for multi-GPU, TPU, or DeepSpeed). For manual optimization, this has no special meaning, as returning the loss is not required.

In this step you’d normally do the forward pass and calculate the loss for a batch. You can also do fancier things like multiple forward passes or something model specific.

Example:

def training_step(self, batch, batch_idx):
    x, y, z = batch
    out = self.encoder(x)
    loss = self.loss(out, x)
    return loss

To use multiple optimizers, you can switch to ‘manual optimization’ and control their stepping:

def __init__(self):
    super().__init__()
    self.automatic_optimization = False


# Multiple optimizers (e.g.: GANs)
def training_step(self, batch, batch_idx):
    opt1, opt2 = self.optimizers()

    # do training_step with encoder
    ...
    opt1.step()
    # do training_step with decoder
    ...
    opt2.step()

Note

When accumulate_grad_batches > 1, the loss returned here will be automatically normalized by accumulate_grad_batches internally.

video(dataloader, epoch=0, set='val')
Parameters

  • dataloader – data loader from train or val set

  • epoch – epoch

  • set – can be either train or val

Returns:

Help function for corner detection

metavision_core_ml.corner_detection.utils.clean_pred(pred, threshold=0.3)

Create a binary mask from a prediction between 0 and 1 after removal of local maximas :param pred: prediction of the network after the sigmoid layer TxBxCxHxW :param threshold: Value of local maximas to consider corners

Returns: Binary mask of corners locations.

metavision_core_ml.corner_detection.utils.events_as_pol(events, frame)

From events creates an image :param events: events psee format :param frame: numpy array to show events on

Returns

updated frame

metavision_core_ml.corner_detection.utils.get_harris_corners_from_image(img, return_mask=False)

takes an image as input and outputs harris corners

Parameters

  • img – opencv image

  • return_mask – returns a binary heatmap instead of corners positions

Returns

harris corners in 3d with constant depth of one or a binary heatmap

metavision_core_ml.corner_detection.utils.numpy_nms(input_array, size=7)

runs non maximal suppression on square patches of size x size on the two last dimension :param input_tensor: numpy array of shape B, C, H, W :param size: size of the side of the square patch for NMS :type size: int

Returns

numpy array where local maximas are unchanged and all other values are -10e5

metavision_core_ml.corner_detection.utils.project_points(points, homography, width, height, original_width, original_height, return_z=False, return_mask=False, filter_correct_corners=True)

projects 2d points given an homography and resize new points to new dimension.

Parameters

  • points – 2d points in the form [x, y, 1] numpy array shape Nx3

  • homography – 3*3 homography numpy array

  • width – desired new dimension

  • height – desired new dimension

  • original_width – original dimension in which homography is given

  • original_height – original dimension in which homography is given

  • return_z – boolean to return points as 2d or 3d

  • return_mask – boolean to return mask of out-of-bounds projected points

  • filter_correct_corners – boolean whether to filter out-of-bounds projected points or not

Returns

points projected in the new space and filtered by default to output only correct points

Return type

projected points

metavision_core_ml.corner_detection.utils.save_ccl_corners(tracker, csv_writer, ts)

Extract corners from the tracker and writes them to a csv :param tracker: ccl tracker class instance :param csv_writer: csv writer :param ts: timestamp of corners

metavision_core_ml.corner_detection.utils.save_nn_corners(tracker, csv_writer, ts)

Extract corners from the tracker and writes them to a csv :param tracker: nearest neighbors tracker class instance :param csv_writer: csv writer :param ts: timestamp of corners

metavision_core_ml.corner_detection.utils.torch_nms(input_tensor, kernel_size=7)

runs non maximal suppression on square patches of size x size on the two last dimension :param input_tensor: torch tensor of shape B, C, H, W :param kernel_size: size of the side of the square patch for NMS :type kernel_size: int

Returns

torch tensor where local maximas are unchanged and all other values are -inf

metavision_core_ml.corner_detection.utils.update_ccl_tracker(tracker, y, x, events_dtype, ts)

Update ccl tracker from torch tensors :param tracker: ccl tracker class instance :param y: torch tensor of corners y positions :param x: torch tensor of corners x positions :param events_dtype: dtype of events :param ts: timestamp of corners

Returns

updated tracker instance

metavision_core_ml.corner_detection.utils.update_nn_tracker(tracker, x, y, ts)

Update nearest neighbors tracker from torch tensors :param tracker: nearest neighbor tracker class instance :param y: torch tensor of corners y positions :param x: torch tensor of corners x positions :param ts: timestamp of corners

Returns

updated tracker instance