SDK Core ML Corner Detection API

functions to compute homography reprojection error from Manderscheid, J., Sironi, A., Bourdis, N., Migliore, D., & Lepetit, V. Speed invariant time surface for learning to detect corner points with event-based cameras. CVPR 2019.

class metavision_core_ml.corner_detection.compute_homography_reprojection_error.ComputeKPI(npy_folder_path)

Computes reprojection error for planar scenes (eg: Atis dataset)

Parameters

  • npy_folder_path – npy_folder_path is the path of the folder containing a .npy file for each sequence to

  • Each file is a list of corners of type (evaluate.) – {‘names’: [‘x’, ‘y’, ‘id’, ‘t’], ‘formats’: [‘<u2’, ‘<u2’, ‘<i4’, ‘<i8’], ‘offsets’: [0, 2, 4, 8], ‘itemsize’: 16}

  • files can be easily created using the function convert_csv_to_npy above. (npy) –

metavision_core_ml.corner_detection.compute_homography_reprojection_error.convert_csv_to_npy(csv_path, overwrite=False)

Converts csv files to npy enable use of EventNpyReader class :param csv_path: path of csv :param overwrite: if csv exists overwrite or not

metavision_core_ml.corner_detection.compute_homography_reprojection_error.homography_error(src_pts, dst_pts)

compute reprojection error between two set of corresponding points by estimating an homography between them :param src_pts: numpy array of 2d corners coordinates in source scene (Nx2) :param dst_pts: numpy array of 2d corners coordinates corresponding supposedly to src_pts in destination scene (Nx2)

Returns

The number of points matched, the reprojection error [0, -1] if no homography was found

metavision_core_ml.corner_detection.compute_homography_reprojection_error.project_coordinates(coordinates, homography)

Using the homography, projects coordinates :param coordinates: 2d numpy array of shape Nx2 :param homography: 3x3 homography matrix

Returns

Projected and normalized coordinates

class to create tracks from corners based on distance in space and time

Image and Corner stream data loader

class metavision_core_ml.corner_detection.corner_video_stream_dataset.CornerVideoDatasetIterator(metadata, height, width, rgb, number_of_heatmaps=10, batch_times=1)

Dataset Iterator streaming images, timestamps and corners

Parameters

  • metadata (object) – path to picture or video

  • height (int) – height of input images / video clip

  • width (int) – width of input images / video clip

  • rgb (bool) – stream rgb videos

  • number_of_heatmaps (int) – The number of heatmaps containing corner locations

  • batch_times (int) – number of timesteps of training sequences

metavision_core_ml.corner_detection.corner_video_stream_dataset.make_corner_video_dataset(path, num_workers, batch_size, height, width, min_length, max_length, number_of_heatmaps=10, rgb=False, seed=None, batch_times=1)

Makes a video/ moving picture dataset.

Parameters

  • path (str) – folder to dataset

  • batch_size (int) – number of video clips / batch

  • height (int) – height

  • width (int) – width

  • min_length (int) – min length of video

  • max_length (int) – max length of video

  • mode (str) – ‘frames’ or ‘delta_t’

  • num_tbins (int) – number of bins in event volume

  • number_of_heatmaps (int) – number of corner heatmaps predicted by the network

  • rgb (bool) – retrieve frames in rgb

  • seed (int) – seed for randomness

  • batch_times (int) – number of time steps in training sequence

metavision_core_ml.corner_detection.corner_video_stream_dataset.pad_collate_fn(data_list)

Here we pad with last image/ timestamp to get a contiguous batch

Here we reuse the GPUSimulator from OpenEB to stream synthetic events.

class metavision_core_ml.corner_detection.data_module.EventToCornerDataModule(hparams)

Simulation gives events + frames + corners

Attributes: prepare_data_per_node:

If True, each LOCAL_RANK=0 will call prepare data. Otherwise only NODE_RANK=0, LOCAL_RANK=0 will prepare data.

allow_zero_length_dataloader_with_multiple_devices:

If True, dataloader with zero length within local rank is allowed. Default value is False.

test_dataloader()

Implement one or multiple PyTorch DataLoaders for testing.

For data processing use the following pattern:

  • download in prepare_data()

  • process and split in setup()

However, the above are only necessary for distributed processing.

Warning

do not assign state in prepare_data

  • test()

  • prepare_data()

  • setup()

Note

Lightning adds the correct sampler for distributed and arbitrary hardware. There is no need to set it yourself.

Returns

A torch.utils.data.DataLoader or a sequence of them specifying testing samples.

Example:

def test_dataloader(self):
    transform = transforms.Compose([transforms.ToTensor(),
                                    transforms.Normalize((0.5,), (1.0,))])
    dataset = MNIST(root='/path/to/mnist/', train=False, transform=transform,
                    download=True)
    loader = torch.utils.data.DataLoader(
        dataset=dataset,
        batch_size=self.batch_size,
        shuffle=False
    )

    return loader

# can also return multiple dataloaders
def test_dataloader(self):
    return [loader_a, loader_b, ..., loader_n]

Note

If you don’t need a test dataset and a test_step(), you don’t need to implement this method.

Note

In the case where you return multiple test dataloaders, the test_step() will have an argument dataloader_idx which matches the order here.

train_dataloader()

Implement one or more PyTorch DataLoaders for training.

Returns

A collection of torch.utils.data.DataLoader specifying training samples. In the case of multiple dataloaders, please see this section.

The dataloader you return will not be reloaded unless you set :paramref:`~pytorch_lightning.trainer.Trainer.reload_dataloaders_every_n_epochs` to a positive integer.

For data processing use the following pattern:

  • download in prepare_data()

  • process and split in setup()

However, the above are only necessary for distributed processing.

Warning

do not assign state in prepare_data

  • fit()

  • prepare_data()

  • setup()

Note

Lightning adds the correct sampler for distributed and arbitrary hardware. There is no need to set it yourself.

Example:

# single dataloader
def train_dataloader(self):
    transform = transforms.Compose([transforms.ToTensor(),
                                    transforms.Normalize((0.5,), (1.0,))])
    dataset = MNIST(root='/path/to/mnist/', train=True, transform=transform,
                    download=True)
    loader = torch.utils.data.DataLoader(
        dataset=dataset,
        batch_size=self.batch_size,
        shuffle=True
    )
    return loader

# multiple dataloaders, return as list
def train_dataloader(self):
    mnist = MNIST(...)
    cifar = CIFAR(...)
    mnist_loader = torch.utils.data.DataLoader(
        dataset=mnist, batch_size=self.batch_size, shuffle=True
    )
    cifar_loader = torch.utils.data.DataLoader(
        dataset=cifar, batch_size=self.batch_size, shuffle=True
    )
    # each batch will be a list of tensors: [batch_mnist, batch_cifar]
    return [mnist_loader, cifar_loader]

# multiple dataloader, return as dict
def train_dataloader(self):
    mnist = MNIST(...)
    cifar = CIFAR(...)
    mnist_loader = torch.utils.data.DataLoader(
        dataset=mnist, batch_size=self.batch_size, shuffle=True
    )
    cifar_loader = torch.utils.data.DataLoader(
        dataset=cifar, batch_size=self.batch_size, shuffle=True
    )
    # each batch will be a dict of tensors: {'mnist': batch_mnist, 'cifar': batch_cifar}
    return {'mnist': mnist_loader, 'cifar': cifar_loader}
val_dataloader()

Implement one or multiple PyTorch DataLoaders for validation.

The dataloader you return will not be reloaded unless you set :paramref:`~pytorch_lightning.trainer.Trainer.reload_dataloaders_every_n_epochs` to a positive integer.

It’s recommended that all data downloads and preparation happen in prepare_data().

  • fit()

  • validate()

  • prepare_data()

  • setup()

Note

Lightning adds the correct sampler for distributed and arbitrary hardware There is no need to set it yourself.

Returns

A torch.utils.data.DataLoader or a sequence of them specifying validation samples.

Examples:

def val_dataloader(self):
    transform = transforms.Compose([transforms.ToTensor(),
                                    transforms.Normalize((0.5,), (1.0,))])
    dataset = MNIST(root='/path/to/mnist/', train=False,
                    transform=transform, download=True)
    loader = torch.utils.data.DataLoader(
        dataset=dataset,
        batch_size=self.batch_size,
        shuffle=False
    )

    return loader

# can also return multiple dataloaders
def val_dataloader(self):
    return [loader_a, loader_b, ..., loader_n]

Note

If you don’t need a validation dataset and a validation_step(), you don’t need to implement this method.

Note

In the case where you return multiple validation dataloaders, the validation_step() will have an argument dataloader_idx which matches the order here.

Defines the network’s architecture used in Detecting Stable Keypoints from Events through Image Gradient Prediction and Long-Lived Accurate Keypoints in Event Streams inspired from Fast Image Reconstruction with an Event Camera

class metavision_core_ml.corner_detection.firenet.FireNet(cin=1, cout=1, base=12)

Initializes internal Module state, shared by both nn.Module and ScriptModule.

forward(x, mask=None)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Here we reuse the GPUSimulator from OpenEB to stream synthetic events and corners.

class metavision_core_ml.corner_detection.gpu_corner_esim.GPUEBSimCorners(dataloader, simulator, batch_times, event_volume_depth, randomize_noises, device, number_of_heatmaps, height, width, batch_size)

Simulated Events on GPU returns events, images and corners

Parameters

  • dataloader – video-clips dataloader

  • simulator – gpu-simulator

  • batch_times – number of rounds per batch

  • event_volume_depth – number of timesteps per round

  • device – hardware to run simulation on

classmethod from_params(folder, num_workers, batch_size, batch_times, event_volume_depth, height, width, min_frames_per_video, max_frames_per_video, number_of_heatmaps, randomize_noises=False, device='cuda:0')

Creates the simulator from parameters :param folder: folder of images :param num_workers: number of workers :param batch_size: size of batch :param batch_times: time dimension per batch :param event_volume_depth: number of channels in event volume :param height: height of images :param width: width of images :param min_frames_per_video: minimum number of frames per video :param max_frames_per_video: maximum number of frames per video :param number_of_heatmaps: number of heatmaps of corners locations returned :param randomize_noises: whether or not to randomize the noise of the simulator :param device: location of data

Returns

GPUEBSimCorners class instantiated

randomize_noises(first_times)

Randomizes noise in the simulator consistent with the batches :param first_times: whether or not the video in the batch is new

metavision_core_ml.corner_detection.gpu_corner_esim.collect_target_images(gray_images, timestamps, video_len, target_indices, num_heatmaps)

Collect target frames + timestamps at target indices and rearranges them into T,B,C,H,W tensor

Parameters

  • gray_images (tensor) – H,W,T format (videos are concatenated along 3rd dimension

  • timestamps (tensor) – B,T

  • video_len (tensor) – B lengths

  • target_indices (tensor) – B,M indices

  • num_heatmaps (int) – number of heatmaps

Pytorch Lightning module

class metavision_core_ml.corner_detection.lightning_model.CornerDetectionCallback(data_module, video_result_every_n_epochs=2)

callbacks to our model

on_train_epoch_end(trainer, pl_module)

Called when the train epoch ends.

To access all batch outputs at the end of the epoch, either:

  1. Implement training_epoch_end in the LightningModule and access outputs via the module OR

  2. Cache data across train batch hooks inside the callback implementation to post-process in this hook.

class metavision_core_ml.corner_detection.lightning_model.CornerDetectionLightningModel(hparams: argparse.Namespace)

Corner Detection: Train your FireNet model to predict corners as a heatmap

configure_optimizers()

Choose what optimizers and learning-rate schedulers to use in your optimization. Normally you’d need one. But in the case of GANs or similar you might have multiple.

Returns

Any of these 6 options.

  • Single optimizer.

  • List or Tuple of optimizers.

  • Two lists - The first list has multiple optimizers, and the second has multiple LR schedulers (or multiple lr_scheduler_config).

  • Dictionary, with an "optimizer" key, and (optionally) a "lr_scheduler" key whose value is a single LR scheduler or lr_scheduler_config.

  • Tuple of dictionaries as described above, with an optional "frequency" key.

  • None - Fit will run without any optimizer.

The lr_scheduler_config is a dictionary which contains the scheduler and its associated configuration. The default configuration is shown below.

lr_scheduler_config = {
    # REQUIRED: The scheduler instance
    "scheduler": lr_scheduler,
    # The unit of the scheduler's step size, could also be 'step'.
    # 'epoch' updates the scheduler on epoch end whereas 'step'
    # updates it after a optimizer update.
    "interval": "epoch",
    # How many epochs/steps should pass between calls to
    # `scheduler.step()`. 1 corresponds to updating the learning
    # rate after every epoch/step.
    "frequency": 1,
    # Metric to to monitor for schedulers like `ReduceLROnPlateau`
    "monitor": "val_loss",
    # If set to `True`, will enforce that the value specified 'monitor'
    # is available when the scheduler is updated, thus stopping
    # training if not found. If set to `False`, it will only produce a warning
    "strict": True,
    # If using the `LearningRateMonitor` callback to monitor the
    # learning rate progress, this keyword can be used to specify
    # a custom logged name
    "name": None,
}

When there are schedulers in which the .step() method is conditioned on a value, such as the torch.optim.lr_scheduler.ReduceLROnPlateau scheduler, Lightning requires that the lr_scheduler_config contains the keyword "monitor" set to the metric name that the scheduler should be conditioned on.

Metrics can be made available to monitor by simply logging it using self.log('metric_to_track', metric_val) in your LightningModule.

Note

The frequency value specified in a dict along with the optimizer key is an int corresponding to the number of sequential batches optimized with the specific optimizer. It should be given to none or to all of the optimizers. There is a difference between passing multiple optimizers in a list, and passing multiple optimizers in dictionaries with a frequency of 1:

  • In the former case, all optimizers will operate on the given batch in each optimization step.

  • In the latter, only one optimizer will operate on the given batch at every step.

This is different from the frequency value specified in the lr_scheduler_config mentioned above.

def configure_optimizers(self):
    optimizer_one = torch.optim.SGD(self.model.parameters(), lr=0.01)
    optimizer_two = torch.optim.SGD(self.model.parameters(), lr=0.01)
    return [
        {"optimizer": optimizer_one, "frequency": 5},
        {"optimizer": optimizer_two, "frequency": 10},
    ]

In this example, the first optimizer will be used for the first 5 steps, the second optimizer for the next 10 steps and that cycle will continue. If an LR scheduler is specified for an optimizer using the lr_scheduler key in the above dict, the scheduler will only be updated when its optimizer is being used.

Examples:

# most cases. no learning rate scheduler
def configure_optimizers(self):
    return Adam(self.parameters(), lr=1e-3)

# multiple optimizer case (e.g.: GAN)
def configure_optimizers(self):
    gen_opt = Adam(self.model_gen.parameters(), lr=0.01)
    dis_opt = Adam(self.model_dis.parameters(), lr=0.02)
    return gen_opt, dis_opt

# example with learning rate schedulers
def configure_optimizers(self):
    gen_opt = Adam(self.model_gen.parameters(), lr=0.01)
    dis_opt = Adam(self.model_dis.parameters(), lr=0.02)
    dis_sch = CosineAnnealing(dis_opt, T_max=10)
    return [gen_opt, dis_opt], [dis_sch]

# example with step-based learning rate schedulers
# each optimizer has its own scheduler
def configure_optimizers(self):
    gen_opt = Adam(self.model_gen.parameters(), lr=0.01)
    dis_opt = Adam(self.model_dis.parameters(), lr=0.02)
    gen_sch = {
        'scheduler': ExponentialLR(gen_opt, 0.99),
        'interval': 'step'  # called after each training step
    }
    dis_sch = CosineAnnealing(dis_opt, T_max=10) # called every epoch
    return [gen_opt, dis_opt], [gen_sch, dis_sch]

# example with optimizer frequencies
# see training procedure in `Improved Training of Wasserstein GANs`, Algorithm 1
# https://arxiv.org/abs/1704.00028
def configure_optimizers(self):
    gen_opt = Adam(self.model_gen.parameters(), lr=0.01)
    dis_opt = Adam(self.model_dis.parameters(), lr=0.02)
    n_critic = 5
    return (
        {'optimizer': dis_opt, 'frequency': n_critic},
        {'optimizer': gen_opt, 'frequency': 1}
    )

Note

Some things to know:

  • Lightning calls .backward() and .step() on each optimizer as needed.

  • If learning rate scheduler is specified in configure_optimizers() with key "interval" (default “epoch”) in the scheduler configuration, Lightning will call the scheduler’s .step() method automatically in case of automatic optimization.

  • If you use 16-bit precision (precision=16), Lightning will automatically handle the optimizers.

  • If you use multiple optimizers, training_step() will have an additional optimizer_idx parameter.

  • If you use torch.optim.LBFGS, Lightning handles the closure function automatically for you.

  • If you use multiple optimizers, gradients will be calculated only for the parameters of current optimizer at each training step.

  • If you need to control how often those optimizers step or override the default .step() schedule, override the optimizer_step() hook.

classmethod load_from_checkpoint(checkpoint_path)

Primary way of loading a model from a checkpoint. When Lightning saves a checkpoint it stores the arguments passed to __init__ in the checkpoint under "hyper_parameters".

Any arguments specified through **kwargs will override args stored in "hyper_parameters".

Parameters

  • checkpoint_path – Path to checkpoint. This can also be a URL, or file-like object

  • map_location – If your checkpoint saved a GPU model and you now load on CPUs or a different number of GPUs, use this to map to the new setup. The behaviour is the same as in torch.load().

  • hparams_file –

    Optional path to a .yaml or .csv file with hierarchical structure as in this example:

    drop_prob: 0.2
dataloader:
    batch_size: 32

    You most likely won’t need this since Lightning will always save the hyperparameters to the checkpoint. However, if your checkpoint weights don’t have the hyperparameters saved, use this method to pass in a .yaml file with the hparams you’d like to use. These will be converted into a dict and passed into your LightningModule for use.

    If your model’s hparams argument is Namespace and .yaml file has hierarchical structure, you need to refactor your model to treat hparams as dict.

  • strict – Whether to strictly enforce that the keys in checkpoint_path match the keys returned by this module’s state dict.

  • **kwargs – Any extra keyword args needed to init the model. Can also be used to override saved hyperparameter values.

Returns

LightningModule instance with loaded weights and hyperparameters (if available).

Note

load_from_checkpoint is a class method. You should use your LightningModule class to call it instead of the LightningModule instance.

Example:

# load weights without mapping ...
model = MyLightningModule.load_from_checkpoint('path/to/checkpoint.ckpt')

# or load weights mapping all weights from GPU 1 to GPU 0 ...
map_location = {'cuda:1':'cuda:0'}
model = MyLightningModule.load_from_checkpoint(
    'path/to/checkpoint.ckpt',
    map_location=map_location
)

# or load weights and hyperparameters from separate files.
model = MyLightningModule.load_from_checkpoint(
    'path/to/checkpoint.ckpt',
    hparams_file='/path/to/hparams_file.yaml'
)

# override some of the params with new values
model = MyLightningModule.load_from_checkpoint(
    PATH,
    num_layers=128,
    pretrained_ckpt_path=NEW_PATH,
)

# predict
pretrained_model.eval()
pretrained_model.freeze()
y_hat = pretrained_model(x)
training_step(batch, batch_nb)

Here you compute and return the training loss and some additional metrics for e.g. the progress bar or logger.

Parameters
Returns

Any of.

  • Tensor - The loss tensor

  • dict - A dictionary. Can include any keys, but must include the key 'loss'

  • None - Training will skip to the next batch. This is only for automatic optimization.

    This is not supported for multi-GPU, TPU, IPU, or DeepSpeed.

In this step you’d normally do the forward pass and calculate the loss for a batch. You can also do fancier things like multiple forward passes or something model specific.

Example:

def training_step(self, batch, batch_idx):
    x, y, z = batch
    out = self.encoder(x)
    loss = self.loss(out, x)
    return loss

If you define multiple optimizers, this step will be called with an additional optimizer_idx parameter.

# Multiple optimizers (e.g.: GANs)
def training_step(self, batch, batch_idx, optimizer_idx):
    if optimizer_idx == 0:
        # do training_step with encoder
        ...
    if optimizer_idx == 1:
        # do training_step with decoder
        ...

If you add truncated back propagation through time you will also get an additional argument with the hidden states of the previous step.

# Truncated back-propagation through time
def training_step(self, batch, batch_idx, hiddens):
    # hiddens are the hidden states from the previous truncated backprop step
    out, hiddens = self.lstm(data, hiddens)
    loss = ...
    return {"loss": loss, "hiddens": hiddens}

Note

The loss value shown in the progress bar is smoothed (averaged) over the last values, so it differs from the actual loss returned in train/validation step.

Note

When accumulate_grad_batches > 1, the loss returned here will be automatically normalized by accumulate_grad_batches internally.

video(dataloader, epoch=0, set='val')
Parameters

  • dataloader – data loader from train or val set

  • epoch – epoch

  • set – can be either train or val

Returns:

Help function for corner detection

metavision_core_ml.corner_detection.utils.clean_pred(pred, threshold=0.3)

Create a binary mask from a prediction between 0 and 1 after removal of local maximas :param pred: prediction of the network after the sigmoid layer TxBxCxHxW :param threshold: Value of local maximas to consider corners

Returns: Binary mask of corners locations.

metavision_core_ml.corner_detection.utils.events_as_pol(events, frame)

From events creates an image :param events: events psee format :param frame: numpy array to show events on

Returns

updated frame

metavision_core_ml.corner_detection.utils.get_harris_corners_from_image(img, return_mask=False)

takes an image as input and outputs harris corners

Parameters

  • img – opencv image

  • return_mask – returns a binary heatmap instead of corners positions

Returns

harris corners in 3d with constant depth of one or a binary heatmap

metavision_core_ml.corner_detection.utils.numpy_nms(input_array, size=7)

runs non maximal suppression on square patches of size x size on the two last dimension :param input_tensor: numpy array of shape B, C, H, W :param size: size of the side of the square patch for NMS :type size: int

Returns

numpy array where local maximas are unchanged and all other values are -10e5

metavision_core_ml.corner_detection.utils.project_points(points, homography, width, height, original_width, original_height, return_z=False, return_mask=False, filter_correct_corners=True)

projects 2d points given an homography and resize new points to new dimension.

Parameters

  • points – 2d points in the form [x, y, 1] numpy array shape Nx3

  • homography – 3*3 homography numpy array

  • width – desired new dimension

  • height – desired new dimension

  • original_width – original dimension in which homography is given

  • original_height – original dimension in which homography is given

  • return_z – boolean to return points as 2d or 3d

  • return_mask – boolean to return mask of out-of-bounds projected points

  • filter_correct_corners – boolean whether to filter out-of-bounds projected points or not

Returns

points projected in the new space and filtered by default to output only correct points

Return type

projected points

metavision_core_ml.corner_detection.utils.save_ccl_corners(tracker, csv_writer, ts)

Extract corners from the tracker and writes them to a csv :param tracker: ccl tracker class instance :param csv_writer: csv writer :param ts: timestamp of corners

metavision_core_ml.corner_detection.utils.save_nn_corners(tracker, csv_writer, ts)

Extract corners from the tracker and writes them to a csv :param tracker: nearest neighbors tracker class instance :param csv_writer: csv writer :param ts: timestamp of corners

metavision_core_ml.corner_detection.utils.torch_nms(input_tensor, kernel_size=7)

runs non maximal suppression on square patches of size x size on the two last dimension :param input_tensor: torch tensor of shape B, C, H, W :param kernel_size: size of the side of the square patch for NMS :type kernel_size: int

Returns

torch tensor where local maximas are unchanged and all other values are -inf

metavision_core_ml.corner_detection.utils.update_ccl_tracker(tracker, y, x, events_dtype, ts)

Update ccl tracker from torch tensors :param tracker: ccl tracker class instance :param y: torch tensor of corners y positions :param x: torch tensor of corners x positions :param events_dtype: dtype of events :param ts: timestamp of corners

Returns

updated tracker instance

metavision_core_ml.corner_detection.utils.update_nn_tracker(tracker, x, y, ts)

Update nearest neighbors tracker from torch tensors :param tracker: nearest neighbor tracker class instance :param y: torch tensor of corners y positions :param x: torch tensor of corners x positions :param ts: timestamp of corners

Returns

updated tracker instance