-
条纹投影三维成像技术通过将立体视觉中一个摄像机替换成光源发生器(如投影仪)而实现,原理如图1所示。光源向被测物体投影按照一定规则和模式编码的图像,形成主动式三维形态测量。编码图案受到物体表面形状的调制而产生形变,而带有形变的结构光被另外位置的相机拍摄到,通过相机投影光源之间的位置关系和结构光形变的程度可以确定出物体的三维形貌。条纹投影技术本质上区别于干涉测量技术,但它采用的条纹形式和干涉测量中两束相干光干涉产生的原理相类似。相比于立体视觉法,其最大优点在于求解物体初相位时是点对点的运算,即在原理上中心点的相位值不受相邻点光强值的影响,从而避免了物面反光率不均匀或观察视角的偏差引起的误差,测量精度可以达到几十或几个微米。
条纹投影技术大体上包含系统标定与三维成像两个方面。系统标定的目的在于获取相机与投影仪的内外参数,为相位与三维坐标转换提供参考系[28-29]。而另一部分三维成像的目的在于通过分析采集的光栅图像,求解相位信息,结合系统标定部分获得的参数进行相位深度之间的转换,完成三维模型重建。文中将简要回顾三维成像部分的基本原理。该部分可细分为三个步骤:条纹分析、相位展开、相位与三维坐标转换。
-
条纹投影技术通常采用正弦条纹图像作为照明图案对被测表面进行编码,采集的条纹图案一般可表示为:
$$ I(x,y) = A(x,y) + B(x,y)\cos \phi (x,y) $$ (1) 式中:
$ (x,y)$ 为像素坐标;$ A$ 为背景光强;$ B$ 为调制度;$ \phi$ 为相位。傅立叶轮廓术[30, 8]是一种常用的条纹分析方法,通过利用带通滤波器提取光栅频谱的正负一级谱,可获得:$$ {I^{\prime} }(x,y) = \frac{1}{2}B(x,y){{\rm{e}}^{i\phi (x,y)}} $$ (2) 随后,利用反正切函数计算相位
$ \phi$ $$ \phi (x,y) = \arctan \frac{{{\mathop{\rm Re}\nolimits} \left[ {{I^\prime }(x,y)} \right]}}{{{\mathop{\rm Im}\nolimits} \left[ {{I^\prime }(x,y)} \right]}} $$ (3) 需要注意的是傅立叶轮廓术是一种基于空间滤波的相位计算方法。尽管效率高,但通常假设被测表面为平滑表面,并且需要投影光栅的空间频率足够高[30]。
相移轮廓术[31]是另一种经常使用的光栅条纹分析法,以使用最为广泛的N步相移法为例,相机拍摄一系列具有
$ 2{\text{π}} /N$ 相对相移的光栅图像$$ {I_n}(x,y) = A(x,y) + B(x,y)\cos [\phi (x,y) + 2{\text{π}}n/N] $$ (4) 式中:
$ n$ 为相移指数($ n = 1,2, \ldots ,N$ )。当拍摄的图像大于三幅时(即$ N \geqslant 3$ ),利用最小二乘法[32],可计算物体相位$ \phi $ :$$ \phi (x,y) = \arctan \frac{{\sum\limits_{n = 1}^N {{I_n}} \sin \left( {\frac{{2{\text{π}}n}}{N}} \right)}}{{\sum\limits_{n = 1}^N {{I_n}} \cos \left( {\frac{{2{\text{π}}n}}{N}} \right)}} $$ (5) 与傅立叶轮廓术相比,相移法的优势在于相位解算精度高。更进一步,随着相移步数的增加,光栅图像的噪声[31]、系统的非线性(如投影仪的Gamma)[33]以及光栅图像的饱和问题[34]对相位计算造成的影响都将减小。
-
无论是傅立叶轮廓术(公式(3)),还是相移轮廓术(公式(5)),解调得到的相位均是包裹相位。它的空间分布是截断的,存在
$ {2{\text{π}}}$ 相位跳变。为了获得连续的真实空间相位分布,需要对其进行相位展开$$ \varPhi (x,y) = \phi (x,y) + 2{\text{π}}k(x,y) $$ (6) 式中:
$ \varPhi (x,y)$ 为去包裹相位或展开相位;$ k(x,y)$ 为光栅条纹的级次。相位展开算法目的在于确定光栅条纹的级次
$ k(x,y)$ 。根据求取条纹级次的原理不同,常见的相位展开方法可以被分为空域展开法[35]与时域展开法两类[36]。空域相位展开是指利用相邻像素的相位值所提供的约束来计算绝对相位值,但该方法依赖于被测物体表面连续的假设。如果被测场景中包含多个孤立物体,或者被测物存在处于不连续表面边界的相邻像素的绝对相位值相差超过$ {2{\text{π}}}$ ,则存在条纹级次歧义,从而无法正确展开。与空间相位展开相比,时间相位展开中每个像素的条纹级次都在时间轴上独立计算,无需参考邻近像素,因此可以展开任意复杂形状表面的包裹相位值。但就相位展开的效率而言,时间相位展开通常还需要至少一幅额外的参考相位图。 -
若将投影仪看做“反相机”来处理,根据双目视觉原理[37],对于相机存在如下投影关系:
$$ {\alpha ^c}{(x,y,1)^{\rm{T}}} = {K^c}\left[ {{R^c},{t^c}} \right]{(X,Y,Z,1)^{\rm{T}}} $$ (7) 对于投影仪存在如下投影关系
$$ {\alpha ^p}{\left( {{x^p},{y^p},1} \right)^{\rm{T}}} = {K^p}\left[ {{R^p},{t^p}} \right]{(X,Y,Z,1)^{\rm{T}}} $$ (8) 将展开后的相位
$ \varPhi (x,y)$ 作为线索,可构建相机坐标与投影仪坐标之间的关系:$$ {x^p} = \frac{{\varPhi (x,y)}}{{2\pi {f_0}}}{w^p} $$ (9) 式中:
$ \alpha $ 为缩放因子;$ K$ 为内参;$ R$ 为旋转矩阵;$ t$ 为平移向量;$ f_0$ 为光栅频率;$ w^p$ 为投影仪分辨率。在预先矫正系统的畸变后,通过联立公式(7)~(9),可获得的相机像素$ {(x,y)}$ 对应的三维坐标$ (X,Y,Z)^T$ 。至此,笔者简要回顾了条纹投影的基本原理。这些基本原理构成了条纹投影技术的物理模型。传统的条纹投影技术是在“物理(模型)”驱动下的技术。下面将介绍通过运用深度学习技术,条纹投影技术也可成为一种在“数据”驱动下的技术,并且在这种情况下,它展现出了超越传统算法的能力。
Application of deep learning technology to fringe projection 3D imaging
-
摘要: 条纹投影(结构光)三维成像是一种广泛使用的三维成像手段。近年来,集成式的三维传感器发展迅速,特别是基于结构光原理的三维传感器件已逐渐成为高端智能手机必不可少的一个重要传感单元。然而随着应用需求的不断增多,人们对条纹投影三维成像这项技术的效率、精度、稳定性等方面的要求也越来越高。同时近年来,深度学习技术的飞速发展已经为光学成像技术的发展开启了一扇新的大门,并且从这扇大门中人们注意到伴随着人工智能概念的引入,条纹投影技术的发展也正在经历着新的突破。首先简要介绍了条纹投影三维成像的基本理论。随后举例分析通过运用深度学习技术,起初基于物理模型的条纹投影技术也可成为一种在“数据”驱动下实现的技术,而且在这种情况下,它展现出了超越传统算法的潜力。最后从神经网络模型、训练数据、训练方法等方面,讨论该领域面临的挑战与未来的研究方向。Abstract: Fringe projection(structured light) 3D imaging is a widely used 3D imaging method. In recent years, the integrated three-dimensional sensor has developed rapidly, especially the three-dimensional sensor based on the principle of structured light has gradually become an essential sensor unit for high-end smart phones. However, with the increasing requirements from applications, people have higher and higher requirements on the efficiency, accuracy, stability and other aspects for the fringe projection technique. At the same time, the rapid development of deep learning technology has opened a new door for the development of optical imaging technology, and from this door we notice that with the introduction of the concept of artificial intelligence, the development of fringe projection technology is also experiencing a new breakthrough. In this paper, the basic theory of fringe projection 3D imaging was introduced. Then, by using the deep learning technology, the fringe projection technology based on the physical model can become a technology driven by "data", and in this case, it showed the potential to surpass the traditional algorithm. Finally, the challenges and future research directions in this field from the aspects of neural network model, training data, training methods and so on were discussed.
-
Key words:
- fringe projection /
- 3D imaging /
- deep learning /
- phase retrieval
-
图 11 动态蜡烛火焰的包裹相位展开结果对比[44]。Wrap表示包裹相位;CNN表示该方法获得的展开相位;LS表示最小二乘法获得的展开相位;Diff为CNN法与LS法计算结果之间的差异
Figure 11. Comparison of results of phase unwrapping of dynamic candle flame[44]. Wrap represents the wrapped phase; CNN represents the phase unwrapped by this method; LS represents the phase unwrapped by the least square method; Diff represents the difference between the results of CNN and LS methods
图 15 针对球面、三角斜面和人脸头像光栅图的实验结果图[46]。第一列为输入神经网络的条纹图;第二列为真实的高度分布;第三列为神经网络输出的高度分布;最后一列为根据第二列与第三列得出的误差分布图
Figure 15. Experimental results of spherical, triangular bevel and face image grating[46]. The first column is the fringe image of the input neural network; the second column is the true simulated height distribution; the third column is the height distribution of the output of the neural network; the last column is the error distribution map based on the second column and the third column
-
[1] Harding K. Industrial metrology: engineering precision [J]. Nature Photonics, 2008, 2(11): 667. doi: 10.1038/nphoton.2008.218 [2] Luhmann T. Close range photogrammetry for industrial applications [J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2010, 65(6): 558−569. doi: 10.1016/j.isprsjprs.2010.06.003 [3] Ma Y, Soatto S, Koseck J, et al. An Invitation to 3-D Vision: from Images to Geometric Models[M]. New York: Springer Science & Business Media, 2012, 26. [4] Jiang H, Zhao H, Li X. High dynamic range fringe acquisition: A novel 3-D scanning technique for high-reflective surfaces [J]. Optics and Lasers in Engineering, 2012, 50(10): 1484−1493. doi: 10.1016/j.optlaseng.2011.11.021 [5] Salvi J, Fernandez S, Pribanic T, et al. A state of the art in structured light patterns for surface profilometry [J]. Pattern Recognition, 2010, 43(8): 2666−2680. doi: 10.1016/j.patcog.2010.03.004 [6] Feng S, Zuo C, Tao T, et al. Robust dynamic 3-D measurements with motion-compensated phase-shifting profilometry [J]. Optics and Lasers in Engineering, 2018, 103: 127−138. [7] Zhang Z H. Review of single-shot 3D shape measurement by phase calculation-based fringe projection techniques [J]. Optics and Lasers in Engineering, 2012, 50(8): 1097−1106. doi: 10.1016/j.optlaseng.2012.01.007 [8] Su X, Zhang Q. Dynamic 3-D shape measurement method: a review [J]. Optics and Lasers in Engineering, 2010, 48(2): 191−204. doi: 10.1016/j.optlaseng.2009.03.012 [9] Wang Y, Liu Z, Jiang C, et al. Motion induced phase error reduction using a Hilbert transform [J]. Optics Express, 2018, 26(26): 34224. doi: 10.1364/OE.26.034224 [10] Feng S, Chen Q, Zuo C, et al. Fast three-dimensional measurements for dynamic scenes with shiny surfaces [J]. Optics Communications, 2017, 382: 18−27. doi: 10.1016/j.optcom.2016.07.057 [11] Heist S, Lutzke P, Schmidt I, et al. High-speed three-dimensional shape measurement using GOBO projection [J]. Optics and Lasers in Engineering, 2016, 87: 90−96. doi: 10.1016/j.optlaseng.2016.02.017 [12] Borowiec S. AlphaGo seals 4-1 victory over Go grandmaster Lee Sedol [J]. The Guardian, 2016: 15. [13] ZˇBONTAR J, Lecun Y. Stereo matching by training a convolutional neural network to compare image patches[J]. The Journal of Machine Learning Research, 2016, 17(1): 2287-2318. [14] Luo W, Schwing A G, Urtasun R. Efficient deep learning for stereo matching[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016: 5695−5703. [15] Li S, Deng M, Lee J, et al. Imaging through glass diffusers using densely connected convolutional networks[J]. arXiv: 1711.06810[physics], 2017. [16] Moriya T, Roth H R, Nakamura S, et al. Unsupervised segmentation of 3D medical images based on clustering and deep representation learning[J]. arXiv: 1804.03830[cs], 2018: 71. [17] Li H, Wei T, Ren A, et al. Deep reinforcement learning: framework, applications, and embedded implementations[J]. arXiv: 1710.03792[cs], 2017. [18] Kuznietsov Y, Stuckler J, Leibe B. Semi-supervised deep learning for monocular depth map prediction[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI: IEEE, 2017: 2215-2223. [19] Kendall A, Grimes M, Cipolla R. PoseNet: A convolutional network for real-time 6-DOF camera relocalization[C]//2015 IEEE International Conference on Computer Vision (ICCV), 2015: 2938−2946. [20] Wang H, Rivenson Y, Jin Y, et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy [J]. Nature Methods, 2019, 16(1): 103−110. doi: 10.1038/s41592-018-0239-0 [21] Rivenson Y, Zhang Y, GÜNAYDIN H, et al. Phase recovery and holographic image reconstruction using deep learning in neural networks [J]. Light: Science & Applications, 2018, 7(2): 17141. [22] Nguyen T, Xue Y, Li Y, et al. Deep learning approach for Fourier ptychography microscopy [J]. Optics Express, 2018, 26(20): 26470. doi: 10.1364/OE.26.026470 [23] Horisaki R, Takagi R, Tanida J. Learning-based imaging through scattering media [J]. Optics Express, 2016, 24(13): 13738. doi: 10.1364/OE.24.013738 [24] Lyu M, Wang W, Wang H, et al. Deep-learning-based ghost imaging [J]. Scientific Reports, 2017, 7(1): 17865. doi: 10.1038/s41598-017-18171-7 [25] Nehme E, Weiss L E, Michaeli T, et al. Deep-STORM: super-resolution single-molecule microscopy by deep learning [J]. Optica, 2018, 5(4): 458−464. doi: 10.1364/OPTICA.5.000458 [26] Fang L, Cunefar D, Wang C, et al. Automatic segmentation of nine retinal layer boundaries in OCT images of non-exudative AMD patients using deep learning and graph search [J]. Biomedical Optics Express, 2017, 8(5): 2732−2744. doi: 10.1364/BOE.8.002732 [27] Li Y, Xue Y, Tian L. Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media [J]. Optica, 2018, 5(10): 1181−1190. doi: 10.1364/OPTICA.5.001181 [28] Zhang S, Huang P S. Novel method for structured light system calibration [J]. Optical Engineering, 2006, 45(8): 083601. doi: 10.1117/1.2336196 [29] Yin Y, Peng X, Li A, et al. Calibration of fringe projection profilometry with bundle adjustment strategy [J]. Optics Letters, 2012, 37(4): 542−544. doi: 10.1364/OL.37.000542 [30] Takeda M, Mutoh K. Fourier transform profilometry for the automatic measurement of 3-D object shapes [J]. Applied Optics, 1983, 22(24): 3977−3982. doi: 10.1364/AO.22.003977 [31] Zuo C, Feng S, Huang L, et al. Phase shifting algorithms for fringe projection profilometry: A review [J]. Optics and Lasers in Engineering, 2018, 109: 23−59. doi: 10.1016/j.optlaseng.2018.04.019 [32] Malacara D. Optical Shop Testing[M]. Hoboken, New Jersey: John Wiley & Sons, 2007, 59. [33] Hoang T, Pan B, Nguyen D, et al. Generic gamma correction for accuracy enhancement in fringe-projection profilometry [J]. Optics Letters, 2010, 35(12): 1992−1994. doi: 10.1364/OL.35.001992 [34] Feng S, Zhang L, Zuo C, et al. High dynamic range 3D measurements with fringe projection profilometry: a review [J]. Measurement Science and Technology, 2018, 29(12): 122001. doi: 10.1088/1361-6501/aae4fb [35] Su X, Chen W. Reliability-guided phase unwrapping algorithm: a review [J]. Optics and Lasers in Engineering, 2004, 42(3): 245−261. doi: 10.1016/j.optlaseng.2003.11.002 [36] Zuo C, Huang L, Zhang M, et al. Temporal phase unwrapping algorithms for fringe projection profilometry: A comparative review [J]. Optics and Lasers in Engineering, 2016, 85: 84−103. doi: 10.1016/j.optlaseng.2016.04.022 [37] Hartley R, Zisserman A. Multiple View Geometry in Computer Vision[M]. Cambridge: Cambridge University Press, 2004: 673. [38] Kemao Q. Two-dimensional windowed Fourier transform for fringe pattern analysis: principles, applications and implementations [J]. Optics and Lasers in Engineering, 2007, 45(2): 304−317. doi: 10.1016/j.optlaseng.2005.10.012 [39] Zhong J, Weng J. Spatial carrier-fringe pattern analysis by means of wavelet transform: wavelet transform profilometry [J]. Applied Optics, 2004, 43(26): 4993−4998. doi: 10.1364/AO.43.004993 [40] Feng S, Chen Q, Gu G, et al. Fringe pattern analysis using deep learning [J]. Advanced Photonics, 2019, 1(2): 1. [41] Shi J, Zhu X, Wang H, et al. Label enhanced and patch based deep learning for phase retrieval from single frame fringe pattern in fringe projection 3D measurement [J]. Optics Express, 2019, 27(20): 28929. doi: 10.1364/OE.27.028929 [42] Yan K, Yu Y, Hu C, et al. Fringe pattern denoising based on deep learning [J]. Optics Communications, 2019, 437: 148−152. doi: 10.1016/j.optcom.2018.12.058 [43] Spoorthi G E, Gorthi S, Gorthi R K S S. PhaseNet: A deep convolutional neural network for two-dimensional phase unwrapping [J]. IEEE Signal Processing Letters, 2019, 26(1): 54−58. doi: 10.1109/LSP.2018.2879184 [44] Wang K, Li Y, Kemao Q, et al. One-step robust deep learning phase unwrapping [J]. Optics Express, 2019, 27(10): 15100. doi: 10.1364/OE.27.015100 [45] Yin W, Chen Q, Feng S, et al. Temporal phase unwrapping using deep learning [J]. Scientific Reports, 2019, 9(1): 20175. doi: 10.1038/s41598-019-56222-3 [46] Van Der Jeught S, Dirckx J J J. Deep neural networks for single shot structured light profilometry [J]. Optics Express, 2019, 27(12): 17091. doi: 10.1364/OE.27.017091 [47] Lv S, Sun Q, Zhang Y, et al. Projector distortion correction in 3D shape measurement using a structured-light system by deep neural networks [J]. Optics Letters, 2020, 45(1): 204−207. doi: 10.1364/OL.45.000204 [48] Zuo C, Tao T, Feng S, et al. Micro Fourier Transform Profilometry (μ FTP): 3D shape measurement at 10,000 frames per second [J]. Optics and Lasers in Engineering, 2018, 102: 70−91. doi: 10.1016/j.optlaseng.2017.10.013 [49] Feng S, Zuo C, Yin W, et al. Micro deep learning profilometry for high-speed 3D surface imaging [J]. Optics and Lasers in Engineering, 2019, 121: 416−427. doi: 10.1016/j.optlaseng.2019.04.020 [50] Zeiler M D, Fergus R. Visualizing and understanding convolutional networks[J]. arXiv: 1311.2901[cs], 2013. [51] Bergstra J, Bengio Y. Random search for hyper-parameter optimization [J]. Journal of Machine Learning Research, 2012, 13(2): 281−305. [52] Finn C, Abbeel P, Levine S. Model-agnostic meta-learning for fast adaptation of deep networks[C]//Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017: 1126−1135. [53] Zoph B, Le Q V. Neural architecture search with reinforcement learning[J]. arXiv preprint arXiv: 1611.01578, 2016. [54] Tan C, Sun F, Kong T, et al. A survey on deep transfer learning[C]//International Conference on Artificial Neural Networks, 2018: 270−279.
计量
- 文章访问数: 5501
- HTML全文浏览量: 1065
- 被引次数: 0