Physical twinning for joint encoding-decoding optimization in computational optics: a review

Bian et al. Light: Science & Applications (2025)14:162 https://doi.org/10.1038/s41377-025-01810-4 www.nature.com/lsa REVIEW ARTICLE Open Access Physical twinning for joint encoding-decoding optimization in computational optics: a review 1234567890():,; 1234567890():,; 1234567890():,; 1234567890():,; Liheng Bian 1,2 ✉ , Xinrui Zhan1, Rong Yan1, Xuyang Chang , Hua Huang3 ✉ and Jun Zhang1 ✉ 1 Abstract Computational optics introduces computation into optics and consequently helps overcome traditional optical limitations such as low sensing dimension, low light throughput, low resolution, and so on. The combination of optical encoding and computational decoding offers enhanced imaging and sensing capabilities with diverse applications in biomedicine, astronomy, agriculture, etc. With the great advance of artificial intelligence in the last decade, deep learning has further boosted computational optics with higher precision and efficiency. Recently, there developed an end-to-end joint optimization technique that digitally twins optical encoding to neural network layers, and then facilitates simultaneous optimization with the decoding process. This framework offers effective performance enhancement over conventional techniques. However, the reverse physical twinning from optimized encoding parameters to practical modulation elements faces a serious challenge, due to the discrepant gap in such as bit depth, numerical range, and stability. In this regard, this review explores various optical modulation elements across spatial, phase, and spectral dimensions in the digital twin model for joint encoding-decoding optimization. Our analysis offers constructive guidance for finding the most appropriate modulation element in diverse imaging and sensing tasks concerning various requirements of precision, speed, and robustness. The review may help tackle the above twinning challenge and pave the way for next-generation computational optics. Introduction Computational optics, which integrates optics with computation, stands out as a powerful technique for high-dimensional optical information acquisition1. In contrast to traditional optical methods that primarily address the human visual perception of “what you see is what you get", computational optics first employs diverse modulation elements to couple highdimensional information (such as spatial, spectral, and semantic dimensions) into a low-dimensional optical field that can be directly measured by existing detectors, referred to as the encoding procedure. Then, such techniques employ algorithms to recover highdimensional information from the measurements, Correspondence: Liheng Bian () or Hua Huang () or Jun Zhang () 1 State Key Laboratory of CNS/ATM & MIIT Key Laboratory of Complex-field Intelligent Sensing, Beijing Institute of Technology, Beijing & Zhuhai, China 2 Yangtze Delta Region Academy of Beijing Institute of Technology (Jiaxing), Jiaxing, China Full list of author information is available at the end of the article These authors contributed equally: Liheng Bian, Xinrui Zhan referred to as the decoding procedure2. Benefiting from the first-encoding-then-decoding mechanism, computational optics surpass traditional optical limitations of low sensing dimension, low light throughput, low resolution, and so on, developed into a significant and competitive field for optical acquisition and reconstruction3. Nowadays, it holds significant value across various fields such as biomedicine, agriculture, and intelligent manufacturing due to its superior and promising performance, even under extreme conditions4,5. With the great advance of artificial intelligence in the last decade, deep learning has further boosted computational optics with higher precision and efficiency. The deep learning technique dates back to the middle of the 20th century, with theoretical proposals of concepts including backpropagation6,7, and single-layer perceptrons8–11, as shown in Fig. 1. Over the last decades, rapid advancements in parallel computation and big data processing enabled its practical realizations and propelled the second wave of artificial intelligence12. The advancements in artificial intelligence have also led to the emergence of © The Author(s) 2025 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. Bian et al. Light: Science & Applications (2025)14:162 a Lensmaker’s formula Gaussian and Newtonian thin lens formulas 17th century imaging R2 d n0 nl Page 2 of 20 R1 d Illumination ψi Collection ψc Image f Computation f f n0 x1 Light Optics source model x2 ψi ψc φ Optics Sensor model model Prior e 1970 Charge-coupled device (CCD) 1943 Mathematical model of biological neuron Object f Image f Measurement ~1990 Computational imaging 1958 Single-layer perceptron 1986 Backpropagation L1 error ~1990 RNN 1989 CNN 2012 Explosion period of artificial intelligence 2006 Deep learning b a h1,1 h2,1 h1,2 h2,2 h1,3 h2,3 Weight adjustment through backpropagation f Image dataset b Inputs c ~2016 Computational imaging based on deep learning O Output d Optics model c Input convolutional layer Pooling layer layer Sensor model pλ S (I ) S (I ) pλ y S (I ) Hidden layers Encoding fully connected layer Output layer 2018 End-to-end optimization for computational imaging Computational image reconstruction y Loss x -1 L x y min y – F (x) x 2 2 +γ x 2 2 Decoding g Optimal encoding parameters Physical ysical twinning twinnin Fig. 1 Historical evolution of computational optics, with the horizontal axis representing time and the vertical axis indicating research advances. a Illustrates the basic formulations of optics. b Presents the concept of backpropagation6,7. c Depicts the development of Convolutional Neural Networks (CNN)9–11. d Shows the computational imaging framework180. e Demonstrates deep learning-based computational reconstruction181. f Shows the demonstration of end-to-end optimization of both optics and image processing14. g Demonstrates an exemplar physical twinning process. Adapted with permission182. Copyright 2024, Springer Nature deep learning-based computational optics as a pivotal technique. A (...truncated)