-
抗散射成像技术可以对复杂环境下人眼无法直接获取目标信息的散射信号进行重建。光子与散射介质相互作用时,其传播方向发生变化,即“散射”。当光信号透过散射介质后,其主要的信号成分为弹道光子和散射光子[1]。弹道光子几乎没有发生散射,包含着成像目标的直接信息;而散射光子由于发生多次散射,相当于散射介质对携带目标信息的光子进行了编码,无法直接获得目标信息。随着散射介质厚度的增加,弹道光子几乎是可以忽略不计的,因此对透过散射介质的光信号进行解码成为实现抗散射成像的关键。为了从散射光信号中解码出目标信息,自适应光学、波前整形、测量光学传输矩阵等传统的抗散射成像算法不断被提出。但是,这些传统算法求解能力和重建效果有限,并不能为后续的实际应用提供理想的帮助。近几年,深度学习以它强大的优化能力受到了人们的广泛关注,并且已经有研究学者将其应用于抗散射成像领域。无论是用深度学习进行端到端的重建,还是将其作为更好的优化算法,都取得了理想的效果,为抗散射成像在实际中的应用奠定了基础。目前基于深度学习的抗散射成像原理主要有三种,分别是从重建传输矩阵、散斑相关成像以及优化问题角度进行解释,如图1所示。
-
透过散射介质的出射光场可以看作是散射介质对携带目标信息的输入光场的编码,可用一个二维矩阵来表征输入光场和出射光场之间的关系,即传输矩阵(Transmission Matrix, TM)[2]。若TM已知,就可以直接通过对TM做简单的反转操作来获得相机视场内任意位置的聚焦或成像。但是TM的测量过程复杂,只能用于固定的散射系统。参考文献[3-4]提出用神经网络学习输入光场和输出光场之间的关系这一过程看作是在求解单个散射介质的TM,D’Arco等人[5]则直接将双层神经网络解释为实际系统的双TM。
当用TM对散射介质表征时,目标透过散射介质的成像过程就可以描述为一个正向模型:
$$ E\left( {x,y} \right) = T\left( {x,y;\xi ,\eta } \right) \times O\left( {\xi ,\eta } \right) $$ (1) 式中:
$E(x,y)$ 为输出图像的复振幅光场;$ O\left( {\xi ,\eta } \right) $ 为输入的原始目标;$ T\left( {x,y;\xi ,\eta } \right) $ 指的是该散射介质的传输矩阵,那么就可以直接通过对TM进行简单的反转操作来获得原始目标的图像,即:$$ \begin{split} \\ O\left( {\xi ,\eta } \right) = {T^{ - 1}}\left( {x,y;\xi ,\eta } \right) \times E\left( {x,y} \right) \end{split} $$ (2) 但是由于浑浊介质的固有复杂性,传输矩阵中的元素之间基本没有联系,从而传统方法的标定过程往往以单个单元进行,十分复杂耗时。而深度学习通过强大的数据特征提取能力实现对TM的准确快速地建模,使散射介质后的目标重建质量更高,为实际应用提供更加准确的目标信息。
-
参考文献[6–8]从散斑相关成像理论[9]的角度对网络由散斑到原始目标的逆向过程的求解进行阐述。散斑相关成像是一种非侵入的抗散射成像方式,在光学记忆效应[10]的基础上,通过计算散斑的自相关提取傅里叶信息来对目标进行重建。由非相干散斑成像原理可知,散斑自相关可近似于目标本身的自相关与背景项之和[9],即:
$$ I \star I = O \star O + C $$ (3) 式中:
$I$ 为散斑强度分布;$O$ 为散射介质后的原始目标;“★”为自相关操作;$C$ 为背景项。基于此,可以充分利用散斑的傅里叶信息来对目标重建:$$ \tilde O = F{T^{ - 1}}\left[ {\sqrt S {{\rm e}^{i\theta }}} \right] $$ (4) 式中:
$\tilde O$ 为重建目标;$S$ 为目标的功率谱;$\theta $ 为目标的傅里叶相位;$F{T^{ - 1}}$ 表示二维傅里叶逆变换。根据维纳辛钦定理,目标的功率谱为其自相关的傅里叶幅值,只需要求解出傅里叶相位即可恢复目标。由公式(1)可知,散斑自相关可以近似等于原始目标的自相关,由散斑自相关可以提取目标的傅里叶信息。由此,计算散斑的自相关作为网络的输入,可建立与原始目标之间的逆向模型:$$ \tilde O = {F^{ - 1}}\left( {I \star I} \right) $$ (5) 式中:
$ {F}^{-1}(·) $ 表示由散斑到原始目标的逆向关系。通过网络提取散斑中原始目标傅里叶信息,完成对傅里叶相位的恢复,可以最终实现透过未知散射介质的强鲁棒性的抗散射成像。 -
深度神经网络具有非线性的特点,可对高度病态的问题求解。基于深度学习的抗散射成像方法将从散斑中求解原始目标的问题看作是一个逆过程的优化求解问题,他们通过卷积神经网络对目标经过不同散射介质所形成的散斑做优化重建[11-12],甚至有的工作可以通过神经网络有效识别和重建出多模光纤的输入[13]。具体来说,从散斑中恢复原始目标可以看作是成像的逆过程,即:
$$ O = {F^{ - 1}}\left( I \right) $$ (6) 式中:
$O$ 表示待测的目标;$I$ 表示通过光学系统获得的输出结果,即散斑图像;$F$ 表示成像过程的正向模型。逆问题的求解可以看作是一个优化问题:$$ O = argmin{\left\| {F\left( O \right) - I} \right\|^2} + \lambda R\left( O \right) $$ (7) 式中:
$R\left( O \right)$ 是正则项。通过网络强大的特征提取能力可以更好地求出正向模型,$ F(·) $ 直接建立输入端与输出端的传输关系,从而实现未知目标的重建。综上所述,以上三种深度建模原理是目前深度抗散射成像技术的主要理论支撑。从光学传输矩阵角度理解,深度模型是通过网络对散射信号进行相位调制并重建TM,一旦建立该系统的传输矩阵,即可由散斑和传输矩阵求解原始目标信息;从散斑相关成像角度理解,深度建模是结合散斑自相关和原始目标自相关的物理联系,充分利用傅里叶信息建立散斑自相关到原始目标的反向模型,由于自相关物理先验或约束的支撑,网络更具导向性。从问题优化角度理解,深度模型建立了输入端和输出端的关系,可以直接由散斑求解原始目标,但是依赖训练数据量以及网络的数据分析能力。这三种理论从原理上解释了深度学习解决散射成像问题的可行性,为不同网络结构解决散射成像问题打下了坚实的物理基础。
Scattering imaging with deep learning: Physical and data joint modeling optimization (invited)
-
摘要: 为了利用被散射的光信号实现成像,越来越多的散射成像方法被提出。其中深度学习以其强大的数据表征和信息提取能力在散射成像领域发挥着重要的作用。相较于传统散射成像方法,基于深度学习的散射成像方法在成像速度、质量、信息维度等方面都有着巨大的优势。但是,模型训练、模型泛化等问题也制约着该方法的发展。因此,越来越多的研究将物理过程与基于数据驱动的方法进行联合建模,利用物理先验指导神经网络优化。相较于单纯的数据驱动方法而言,物理-数据联合建模的方法对数据量、神经网络参数量的依赖程度大大降低,在保证成像质量的前提下有效降低数据获取难度及对实验环境的要求。联合建模优化的方式实现了介质、目标类型等散射成像中关键节点的泛化。同时在训练过程方面,实现了从有监督到半监督再到无监督的训练优化过程迭代,不同模型和监督方式的提出大大提升了基于深度学习方法的训练效率,在降低对硬件和时间成本的同时,提升了基于深度学习的散射成像方法在非实验室场景应用的可能性。Abstract: More scattering imaging methods have been proposed to realize imaging using scattered optical signals. Deep learning plays an important role in the field of imaging through scattering medium with its powerful data representation ability and information extraction ability. Compared with traditional scattering imaging methods, deep learning-based scattering imaging methods have great advantages in imaging speed, imaging quality, information dimension, and other aspects. However, the problems of model training, model generalization also restrict the development of this method. Therefore, more and more studies jointly model physical processes with data-driven-based methods and use physical priors to guide neural network optimization. Compared with the simple data-driven method, the physical-data joint modeling method greatly reduces the dependence on the amount of data and the number of neural network parameters, which can effectively reduce the difficulty of data acquisition and the requirements for experimental environment under the premise of ensuring the imaging quality. The joint modeling optimization method realizes the generalization of the medium and the type of hidden targets. At the same time, the training strategy of those methods is also being optimized which is realized from the supervised to semi-supervised and then to unsupervised. The proposed different models and supervision strategies greatly improve training efficiency. Those advantages improve the method of imaging through scattering medium based on the deep learning scenario application possibility out of the laboratory while reducing the cost of hardware and time.
-
Key words:
- scattering imaging /
- deep learning /
- computational imaging /
- neural network
-
图 3 基于CNN和GAN的模型优化:(a)解码器输出双通道进行分类[3];(b)解码器采用分支结构提取双目标信息[20];(c)编码器底部分支提取不同尺寸特征[23];(d)解码器增加注意力机制优化上采样特征权重[25]
Figure 3. Optimization of models based on CNN and GAN: (a) The decoder outputs dual channels for classification[3]; (b) The decoder uses a branch structure to extract dual-target information[20]; (c) The bottom branch of the encoder extracts features of different sizes[23]; (d) The decoder increases attention mechanism to optimize upsampling feature weights[25]
图 6 不同抗散射成像方法中的神经网络框架。(a) 单帧彩色目标成像方法[7];(b)透过大光学厚度散射介质成像方法[12];(c)单帧大视场复杂目标成像方法[29];(d)利用神经网络实现基于波前调制透过散射介质成像[5]
Figure 6. Neural network framework in different anti-scattering imaging methods. (a) Single-frame color target imaging method[7]; (b) Imaging method through scattering media of large optical thickness[12]; (c) Imaging method for complex objects with large field of view using a single frame speckle[29]; (d) Imaging through scattering medium based on wavefront modulation realized by neural network[5]
图 8 基于物理模型与数据模型联合的感知学习方法范例。(a) 融合ME透过未知介质抗散射成像方法[6];(b) 散斑自相关约束抗散射目标域泛化成像方法[24];(c) 物理残差优化的三维立体感知物理学习方法[51];(d) 物理学习设计编码照明硬件优化的定量相位成像方法[52]
Figure 8. Examples of the physics-aware learning framework combined with physical model and data model. (a) Imaging through unknown diffusers via fusing the ME prior [6]; (b) Imaging unknown target through scattering media via autocorrelation constraint [24]; (c) Untrained deep learning-based 3 D sensing method via residual physics optimization [51]; (d) Optimized coded-illumination for quantitative phase imaging via Physics-based learning method [52]
-
[1] Zhan X, Gao J, Gan Y, et al. Roles of scattered and ballistic photons in imaging through scattering media: A deep learning-based study [J]. arXiv, 2022: 2207.10263. doi: 10.48550/arXiv.2207.10263 [2] Popoff S M, Lerosey G, Carminati R, et al. Measuring the transmission matrix in optics: An approach to the study and control of light propagation in disordered media [J]. Physical Review Letters, 2010, 104(10): 100601. doi: 10.1103/PhysRevLett.104.100601 [3] Li Y, Xue Y, Tian L. Deep speckle correlation: A deep learning approach toward scalable imaging through scattering media [J]. Optica, 2018, 5(10): 1181-1190. doi: 10.1364/OPTICA.5.001181 [4] Matthès M W, Bromberg Y, de Rosny J, et al. Learning and avoiding disorder in multimode fibers [J]. Physical Review X, 2021, 11(2): 021060. doi: 10.1103/PhysRevX.11.021060 [5] d’Arco A, Xia F, Boniface A, et al. Physics-based neural network for non-invasive control of coherent light in scattering media [J]. Optics Express, 2022, 30(17): 30845-30856. doi: 10.1364/OE.465702 [6] Zhu S, Guo E, Gu J, et al. Imaging through unknown scattering media based on physics-informed learning [J]. Photonics Research, 2021, 9(5): B210-B219. doi: 10.1364/PRJ.416551 [7] Zhu S, Guo E, Gu J, et al. Efficient color imaging through unknown opaque scattering layers via physics-aware learning [J]. Optics Express, 2021, 29(24): 40024-40037. doi: 10.1364/OE.441326 [8] Liao M, Zheng S, Pan S, et al. Deep-learning-based ciphertext-only attack on optical double random phase encryption [J]. Opto-Electronic Advances, 2021, 4(5): 05200016. [9] Katz O, Heidmann P, Fink M, et al. Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations [J]. Nature Photonics, 2014, 8(10): 784-790. doi: 10.1038/nphoton.2014.189 [10] Freund I, Rosenbluh M, Feng S. Memory effects in propagation of optical waves through disordered media [J]. Physical Review Letters, 1988, 61(20): 2328-2331. doi: 10.1103/PhysRevLett.61.2328 [11] Li S, Deng M, Lee J, et al. Imaging through glass diffusers using densely connected convolutional networks [J]. Optica, 2018, 5(7): 803-813. doi: 10.1364/OPTICA.5.000803 [12] Lyu M, Wang H, Li G, et al. Learning-based lensless imaging through optically thick scattering media [J]. Advanced Photonics, 2019, 1(3): 036002. [13] Borhani N, Kakkava E, Moser C, et al. Learning to see through multimode fibers [J]. Optica, 2018, 5(8): 960-966. doi: 10.1364/OPTICA.5.000960 [14] Rahmani B, Loterie D, Konstantinou G, et al. Multimode optical fiber transmission with a deep learning network [J]. Light: Science & Applications, 2018, 7(1): 1-11. doi: 10.1038/s41377-018-0074-1 [15] Wang Z, Jin X, Dai Q. Non-invasive imaging through strongly scattering media based on speckle pattern estimation and deconvolution [J]. Scientific Reports, 2018, 8(1): 9088. doi: 10.1038/s41598-018-27467-1 [16] Cheng S, Li H, Luo Y, et al. Artificial intelligence-assisted light control and computational imaging through scattering media [J]. Journal of Innovative Optical Health Sciences, 2019, 12(4): 1930006. doi: 10.1142/S1793545819300064 [17] Sun Y, Shi J, Sun L, et al. Image reconstruction through dynamic scattering media based on deep learning [J]. Optics Express, 2019, 27(11): 16032-16046. doi: 10.1364/OE.27.016032 [18] Yang M, Liu Z H, Cheng Z D, et al. Deep hybrid scattering image learning [J]. Journal of Physics D: Applied Physics, 2019, 52(11): 115105. doi: 10.1088/1361-6463/aafa3c [19] Yamazaki K, Horisaki R, Tanida J. Imaging through scattering media based on semi-supervised learning [J]. Applied Optics, 2020, 59(31): 9850-9854. doi: 10.1364/AO.402428 [20] Lai X, Li Q, Chen Z, et al. Reconstructing images of two adjacent objects passing through scattering medium via deep learning [J]. Optics Express, 2021, 29(26): 43280-43291. doi: 10.1364/OE.446630 [21] Luo Y, Yan S, Li H, et al. Towards smart optical focusing: Deep learning-empowered dynamic wavefront shaping through nonstationary scattering media [J]. Photonics Research, 2021, 9(8): B262-B278. doi: 10.1364/PRJ.415590 [22] Sun Y, Wu X, Zheng Y, et al. Scalable non-invasive imaging through dynamic scattering media at low photon flux [J]. Optics and Lasers in Engineering, 2021, 144: 106641. doi: 10.1016/j.optlaseng.2021.106641 [23] Zheng S, Wang H, Dong S, et al. Incoherent imaging through highly nonstatic and optically thick turbid media based on neural network [J]. Photonics Research, 2021, 9(5): B220-B228. doi: 10.1364/PRJ.416246 [24] Shi Y, Guo E, Bai L, et al. Prior-free imaging unknown target through unknown scattering medium [J]. Optics Express, 2022, 30(10): 17635-17651. doi: 10.1364/OE.453695 [25] Song B, Jin C, Wu J, et al. Deep learning image transmission through a multimode fiber based on a small training dataset [J]. Optics Express, 2022, 30(4): 5657-5672. doi: 10.1364/OE.450999 [26] Tahir W, Wang H, Tian L. Adaptive 3D descattering with a dynamic synthesis network [J]. Light: Science & Applications, 2022, 11(1): 42. doi: 10.1038/s41377-022-00730-x [27] Turpin A, Vishniakou I, d Seelig J. Light scattering control in transmission and reflection with neural networks [J]. Optics Express, 2018, 26(23): 30911-30929. doi: 10.1364/OE.26.030911 [28] Kang I, Pang S, Zhang Q, et al. Recurrent neural network reveals transparent objects through scattering media [J]. Optics Express, 2021, 29(4): 5316-5326. doi: 10.1364/OE.412890 [29] Guo E, Zhu S, Sun Y, et al. Learning-based method to reconstruct complex targets through scattering medium beyond the memory effect [J]. Optics Express, 2020, 28(2): 2433-2446. doi: 10.1364/OE.383911 [30] Cheng Q, Bai L, Han J, et al. Super-resolution imaging through the diffuser in the near-infrared via physically-based learning [J]. Optics and Lasers in Engineering, 2022, 159: 107186. doi: 10.1016/j.optlaseng.2022.107186 [31] Guo E, Shi Y, Bai L, et al. Imaging complex targets through a scattering medium based on adaptive encoding [J]. Photonics, 2022, 9(7): 467. doi: 10.3390/photonics9070467 [32] Cheng Q, Guo E, Gu J, et al. De-noising imaging through diffusers with autocorrelation [J]. Applied Optics, 2021, 60(25): 7686-7695. doi: 10.1364/AO.425099 [33] Zhu S, Guo E, Cui Q, et al. Locating and imaging through scattering medium in a large depth [J]. Sensors, 2020, 21(1): 90. doi: 10.3390/s21010090 [34] Satat G, Tancik M, Gupta O, et al. Object classification through scattering media with deep learning on time resolved measurement [J]. Optics Express, 2017, 25(15): 17466-17479. doi: 10.1364/OE.25.017466 [35] Caramazza P, Boccolini A, Buschek D, et al. Neural network identification of people hidden from view with a single-pixel, single-photon detector [J]. Scientific Reports, 2018, 8(1): 11945. doi: 10.1038/s41598-018-30390-0 [36] Zhao Q, Li H, Yu Z, et al. Speckle-based optical cryptosystem and its application for human face recognition via deep learning [J]. arXiv, 2022: 2201.11844. doi: 10.48550/arXiv.2201.11844 [37] Kürüm U, Wiecha P R, French R, et al. Deep learning enabled real time speckle recognition and hyperspectral imaging using a multimode fiber array [J]. Optics Express, 2019, 27(15): 20965-20979. doi: 10.1364/OE.27.020965 [38] Guo E, Sun Y, Zhu S, et al. Single-shot color object reconstruction through scattering medium based on neural network [J]. Optics and Lasers in Engineering, 2021, 136: 106310. doi: 10.1016/j.optlaseng.2020.106310 [39] Gao Y, Xu W, Chen Y, et al. Deep learning-based photoacoustic imaging of vascular network through thick porous media [J]. IEEE Transactions on Medical Imaging, 2022, 41(8): 2191-2204. doi: 10.1109/TMI.2022.3158474 [40] Li Q, Zhao J, Zhang Y, et al. Imaging reconstruction through strongly scattering media by using convolutional neural networks [J]. Optics Communications, 2020, 477: 126341. doi: 10.1016/j.optcom.2020.126341 [41] Sun L, Shi J, Wu X, et al. Photon-limited imaging through scattering medium based on deep learning [J]. Optics Express, 2019, 27(23): 33120-33134. doi: 10.1364/OE.27.033120 [42] Li X, Shi J, Wu X, et al. Photon limited imaging through disordered media: information extraction by exploiting the photon’s quantum nature via deep learning [J]. Applied Physics B, 2022, 128(2): 1-13. [43] Han J, Miao J, Shi Y, et al. Photon-limited imaging through scattering medium based on speckle coding [J]. Optik, 2022, 255: 168643. doi: 10.1016/j.ijleo.2022.168643 [44] Liu K, Zhang H, Zhang B, et al. Hybrid optimization algorithm based on neural networks and its application in wavefront shaping [J]. Optics Express, 2021, 29(10): 15517-15527. doi: 10.1364/OE.424002 [45] Zhao J, Sun Y, Zhu H, et al. Deep-learning cell imaging through anderson localizing optical fiber [J]. Advanced Photonics, 2019, 1(6): 066001. [46] Fan P, Zhao T, Su L. Deep learning the high variability and randomness inside multimode fibers [J]. Optics Express, 2019, 27(15): 20241-20258. doi: 10.1364/OE.27.020241 [47] Wu H, Meng X, Yang X, et al. Single shot real-time high-resolution imaging through dynamic turbid media based on deep learning [J]. Optics and Lasers in Engineering, 2022, 149: 106819. doi: 10.1016/j.optlaseng.2021.106819 [48] Barbastathis G, Ozcan A, Situ G. On the use of deep learning for computational imaging [J]. Optica, 2019, 6(8): 921-943. doi: 10.1364/OPTICA.6.000921 [49] Rosen J, de Aguiar H B, Anand V, et al. Roadmap on chaos-inspired imaging technologies (CI2-Tech) [J]. Applied Physics B, 2022, 128(3): 1-26. [50] Ba Y, Zhao G, Kadambi A. Blending diverse physical priors with neural networks [J]. arXiv, 2019: 1910.00201. doi: 10.48550/arXiv.1910.00201 [51] Yu H, Han B, Bai L, et al. Untrained deep learning-based fringe projection profilometry [J]. APL Photonics, 2022, 7(1): 016102. doi: 10.1063/5.0069386 [52] Kellman M R, Bostan E, Repina N A, et al. Physics-based learned design: optimized coded-illumination for quantitative phase imaging [J]. IEEE Transactions on Computational Imaging, 2019, 5(3): 344-353. doi: 10.1109/TCI.2019.2905434 [53] Wu H, Li Q, Meng X, et al. Cryptographic analysis on an optical random-phase-encoding cryptosystem for complex targets based on physics-informed learning [J]. Optics Express, 2021, 29(21): 33558-33571. doi: 10.1364/OE.441293 [54] Stewart R, Ermon S. Label-free supervision of neural networks with physics and domain knowledge[C]//Thirty-First AAAI Conference on Artificial Intelligence, 2017. [55] Pan J, Dong J, Liu Y, et al. Physics-based generative adversarial models for image restoration and beyond [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 43(7): 2449-2462. doi: 10.1109/TPAMI.2020.2969348 [56] Metzler C A, Heide F, Rangarajan P, et al. Deep-inverse correlography: Towards real-time high-resolution non-line-of-sight imaging [J]. Optica, 2020, 7(1): 63-71. doi: 10.1364/OPTICA.374026 [57] Wang F, Bian Y, Wang H, et al. Phase imaging with an untrained neural network [J]. Light: Science & Applications, 2020, 9(1): 1-7. doi: 10.1038/s41377-020-0302-3 [58] Monakhova K, Tran V, Kuo G, et al. Untrained networks for compressive lensless photography [J]. Optics Express, 2021, 29(13): 20913-20929. doi: 10.1364/OE.424075 [59] Diamond S, Sitzmann V, Heide F, et al. Unrolled optimization with deep priors [J]. arXiv, 2017: 1705.08041. doi: 10.48550/arXiv.1705.08041 [60] Li R, Cheong L F, Tan R T. Heavy rain image restoration: Integrating physics model and conditional adversarial learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 1633-1642.