Real-time low-light video enhancement on smartphones

Journal of Real-Time Image Processing (2024) 21:155 https://doi.org/10.1007/s11554-024-01532-7 RESEARCH Real‑time low‑light video enhancement on smartphones Yiming Zhou1 · Callen MacPhee1 · Wesley Gunawan1 · Ali Farahani1 · Bahram Jalali1 Received: 6 June 2024 / Accepted: 30 July 2024 © The Author(s) 2024 Abstract Real-time low-light video enhancement on smartphones remains an open challenge due to hardware constraints such as limited sensor size and processing power. While night mode cameras have been introduced in smartphones to acquire highquality images in light-constrained environments, their usability is restricted to static scenes as the camera must remain stationary for an extended period to leverage long exposure times or burst imaging techniques. Concurrently, significant process has been made in low-light enhancement on images coming out from the camera’s image signal processor (ISP), particularly through neural networks. These methods do not improve the image capture process itself; instead, they function as post-processing techniques to enhance the perceptual brightness and quality of captured imagery for display to human viewers. However, most neural networks are computationally intensive, making their mobile deployment either impractical or requiring considerable engineering efforts. This paper introduces VLight, a novel single-parameter low-light enhancement algorithm that enables real-time video enhancement on smartphones, along with real-time adaptation to changing lighting conditions and user-friendly fine-tuning. Operating as a custom brightness-booster on digital images, VLight provides realtime and device-agnostic enhancement directly on users’ devices. Notably, it delivers real-time low-light enhancement at up to 67 frames per second (FPS) for 4K videos locally on the smartphone. Keywords Real-time low-light enhancement · Video enhancement · Smartphones · Mobile devices 1 Introduction Acquiring high-quality images and videos on smartphones in low-light environments poses significant challenges due to various physical constraints. For instance, small lens apertures limit the number of collected photons, leading to noisy images in low-light conditions. Additionally, compact sensor sizes restrict the number of electrons each pixel can store, * Bahram Jalali Yiming Zhou Callen MacPhee Wesley Gunawan Ali Farahani 1 Electrical and Computer Engineering Department, University of California, Los Angeles, 420 Westwood Plaza, Los Angeles, CA 90095, USA resulting in a limited dynamic range. These factors contribute to a very low signal-to-noise ratio (SNR) when there is insufficient light, significantly degrading image and video quality. To address these issues, modern smartphones now feature night mode in their camera settings, which captures multiple frames with different exposure levels. This allows more light to reach the sensor and effectively reduces noise. Computational imaging approaches are then used to fuse the captured frames by aligning and merging them, followed by local tone mapping to generate a final photo that is brighter, crisper, and has a higher SNR. While effective for static images, night mode is not suitable for live video captures due to the long exposure and extended processing time for each frame. Meanwhile, the demand for real-time low-light video enhancement on mobile devices is growing given the prevalence of these devices as computing platforms and user interfaces. For example, as people increasingly rely on their smartphones for video calls and live streaming, video quality can be compromised in limited lighting conditions. Similarly, videos captured by drones are streamed to smartphones, but the camera may not provide satisfying video Vol.:(0123456789) 155 Page 2 of 15 quality at night. Moreover, processing locally on the device reduces bandwidth usage and enhances privacy by avoiding network transmission. Therefore, enabling real-time night mode for videos on smartphones has recently emerged as an active area of research [1, 2]. Beyond improving image acquisition systems, substantial progress has been made in low-light enhancement of images post-ISP processing. These methods focus on enhancing the perceptual brightness and qualities of captured imagery for display. Classical algorithms based on Retinex theory [3–5], histogram equalization [6], Gamma Correction [7, 8], and their variants have been widely used in practice. However, these algorithms often rely on handcrafted rules and assumptions, which can limit the adaptability to diverse image characteristics. More recently, deep learning approaches utilizing neural networks such as CNNs [9–14] and Vision Transformers [15, 16] have emerged as a revolutionary paradigm in low-light enhancement. Despite their success, the evaluation and benchmarking of these neural networks typically occur in ideal setups with powerful GPU servers and curated datasets. Moreover, their high complexity makes them impractical for applications on platforms with constrained memory and computational resources. Meanwhile, limited attention has been directed towards the development and evaluation of low-light enhancement techniques directly on mobile devices. In this paper, we introduce VLight, a novel single-parameter low-light enhancement algorithm designed to run in real-time on smartphones. VLight is a post-processing algorithm that functions as a custom brightness-boosting curve on digital images coming out from the camera’s image signal processor (ISP). Featuring low complexity and high efficiency, it is well-suited for applications on mobile devices with constrained resources, offering real-time adaptation to changing lighting conditions and user-friendly fine-tuning. Notably, our proposed algorithm delivers real-time low-light video enhancement locally on the smartphone at up to 67 frames per second (FPS) at 4K resolution, which is, to the best of our knowledge, the fastest recorded performance. The main contributions of the present paper are summarized as follows: 1. We introduce VLight, a novel single-parameter low-light enhancement algorithm that features low complexity and high efficiency. VLight’s unique design allows for userfriendly fine-tuning and real-time adaptation to changing lighting conditions. 2. We demonstrate real-time video enhancement on mobile platforms including smartphones and the NVIDIA Jetson Nano. The demonstration showcases VLight’s versatility across various applications, from improving usercaptured visual content on smartphones to enhancing nighttime driving visibility through edge computing. Journal of Real-Time Image Processing (2024) 21:155 The rest of the paper is organized as follows: Sect. 2 discusses related work on low-light enhancement, including image acquisition techniques during the capture and postprocessing methods for captured imagery. Section 3 provides a thorough analysis of VLight’s mathematical principles and development process. (...truncated)