Energy-Efficient Adaptive 3D Sensing

CVPR 2023

Brevin Tilmon¹     Zhanghao Sun²     Sanjeev Koppal¹     Yicheng Wu³
    Georgios Evangelidis³     Ramzi Zahreddine³     Guru Krishnan³     Sizhuo Ma³     Jian Wang³

¹University of Florida     ²Stanford University     ³Snap Inc.

Abstract

Active depth sensing achieves robust depth estimation but is usually limited by the sensing range. Naively increasing the optical power can improve sensing range but induces eye-safety concerns for many applications, including autonomous robots and augmented reality. In this paper, we propose an adaptive active depth sensor that jointly optimizes range, power consumption, and eye-safety. The main observation is that we need not project light patterns to the entire scene but only to small regions of interest where depth is necessary for the application and passive stereo depth estimation fails. We theoretically compare this adaptive sensing scheme with other sensing strategies, such as full-frame projection, line scanning, and point scanning. We show that, to achieve the same maximum sensing distance, the proposed method consumes the least power while having the shortest (best) eye-safety distance. We implement this adaptive sensing scheme with two hardware prototypes, one with a phase-only spatial light modulator (SLM) and the other with a micro-electro-mechanical (MEMS) mirror and diffractive optical elements (DOE). Experimental results validate the advantage of our method and demonstrate its capability of acquiring higher quality geometry adaptively.

Method

We implement our adaptive 3D sensing framework in hardware as an active stereo system. In order to compare against existing 3D sensing techniques used in devices like Intel RealSense and Episcan3D, we emulate these devices on the spatial light modulator. See our code for our CUDA simulator of all 3D sensors. A spatial light modulator acts as a software-defined 3D sensor, enabling us to emulate the light redistribution properties of existing 3D sensors. We then adjust the camera exposure to finish the comparison. The image and video below demonstrates each emualted 3D sensor with the spatial light modulator. We scale the intensities for visualization.

Results

Below we show comparisons of our adaptive sensor versus line scanning and full frame sensors. Notice the dots are much brighter for the adaptive sensor which results in better active stereo depth performance. We also show outdoor results under direct sunlight. Despite the very strong ambient light from the sun, we are able to adaptively reconstruct depth with our sensor. We can rely on passive stereo to reconstruct depth for the majority of the scene, and then fill in remaining missing depths in a small ROI with our adaptive sensor.

Real Time Demo

In this real time adaptive active stereo demo the textureless objects are detected and then adaptively illuminated by the SLM. Notice the depth quality difference when the laser is off versus on. Please see our code for our CUDA-accelerated hologram generator. The demo runs self-contained on an embedded NVIDIA Jetson Nano with a 256 core GPU. Past a certain number of hologram points it is faster to just do Fourier Holography instead of our Fresnel Holography, but our Fresnel approach has significant speed ups for sparse holograms since the FFT runtime is constant for any number of dots and depends on the hologram size instead.

Citation

  @inproceedings{tilmon2023e3d,
    title     = {Energy-Efficient Adaptive 3D Sensing},
    author    = {Tilmon, Brevin and Sun, Zhanghao and Koppal, Sanjeev and Wu, Yicheng and Evangelidis, Georgios and Zahreddine, Ramzi and Krishnan, Guru and Ma, Sizhuo and Wang, Jian},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
  }