-
局部对比度信息[19]是目标信息和背景信息之间的对比,它能够更好地突出目标并且抑制背景,使得目标在检测过程中更加突出。文中利用局部对比度的机理,提出了基于偏振权重的偏振度显著图。
光波的任意偏振态可以由斯托克斯矢量来表示,斯托克斯矢量的表达式如下:
$$ S = \left( \begin{gathered} {S_0} \hfill \\ {S_1} \hfill \\ {S_2} \hfill \\ {S_3} \hfill \\ \end{gathered} \right) = \left( \begin{gathered} \frac{1}{2}({I_0} + {I_{45}} + {I_{90}} + {I_{135}}) \\ {I_0} - {I_{90}} \\ {I_{45}} - {I_{135}} \\ {I_R} - {I_L} \\ \end{gathered} \right) $$ (1) 式中:
${I_0}$ 、${I_{45}}$ 、${I_{90}}$ 、${I_{135}}$ 分别表示偏振方向为0°、45 °、90°、135°的图像强度;${S_0}$ 表示图像的总强度;${S_1}$ 为0°偏振图像与90°偏振图像的强度差,表示水平和垂直方向上的线偏振光分量;${S_2}$ 为45°偏振图像与135°偏振图像的强度差,表示45°或−45°方向上的线偏振光分量;${S_3}$ 为左旋、右旋偏振图像的强度差,表示圆偏振光分量;由于对地观测过程中,圆偏振光分量强度较小,可近似忽略为零。基于上述的斯托克斯矢量,可以计算出偏振度(Degree of polarization,Dop),其表达式如下:
$$ Dop = \frac{{\sqrt {{S_1}^2 + {S_2}^2} }}{{{S_0}}} $$ (2) 偏振度显著图是通过局部对比度计算来得到的。局部对比度的计算是通过滑窗操作来增强滑窗中心的目标区域。将滑窗均匀分割为3×3的区域。在每个滑窗区域中,9个区域分别记为
${R_i}(i = 0,1,2,\cdots,8)$ 。中心区域${R_0}$ 中偏振度最大值为:$$ {L_{Dop}} = \max (I_{Dop}^{{R_0}}) $$ (3) 式中:
${L_{Dop}}$ 表示滑窗区域的中心区域${R_0}$ 的偏振度最大值;$ I_{Dop}^{{R_0}} $ 表示滑窗区域的中心区域${R_0}$ 的每个像素的偏振度值。滑窗区域中每个区域的偏振度平均值
$m_{Dop}^i(i = 0,1,2,\cdots,8)$ 定义如下:$$ m_{Dop}^i = \frac{1}{N}\sum\limits_1^N {I_{Dop}^{{R_i}}} $$ (4) 式中:
$ I_{Dop}^{{R_i}} $ 表示滑窗区域中区域${R_i}$ 的每个像素的偏振度值;$N$ 表示每个区域${R_i}$ 中的像素总数。为了对目标区域进一步增强,引入中心区域的偏振度与背景区域偏振度的差值作为局部对比度的权重,该权重
${\omega _i}(i = 1,2,\cdots,8)$ 表达式如下:$$ {\omega _i} = \left| {m_{Dop}^0 - m_{Dop}^i} \right| $$ (5) 对偏振权重
${\omega _i}(i = 1,2,\cdots,8)$ 进行归一化并求取均值得到归一化后的偏振权重${\omega _p}$ ,表达式如下:$$ {\omega _p} = \frac{1}{8}\sum\limits_{i = 1}^8 {\frac{{{\omega _i} - {\omega _{\min }}}}{{{\omega _{\max }} - {\omega _{\min }}}}} $$ (6) 式中:
${\omega _{\max }}$ 和${\omega _{\min }}$ 分别表示公式(5)计算出的局部滑窗区域权重的最大值和最小值。因此,引入偏振权重的滑窗区域的偏振度局部显著图表示为:
$$ {C_{Dop}} = {\omega _p} \cdot \mathop {\min }\limits_i \frac{{{L_{Dop}}m_{Dop}^0}}{{m_{Dop}^i}},(i = 1,2,\cdots,8) $$ (7) 对整个图像进行遍历之后,得到整个图像的偏振度显著图,通过显著图可以快速关注到较为显著的目标。文中利用自适应阈值操作来对目标与背景进行分割,偏振度显著图的阈值
$T{h_{Dop}}$ 定义如下:$$ T{h_{Dop}} = {\mu _{Dop}} + k{\sigma _{Dop}} $$ (8) 式中:
${\mu _{Dop}} $ 表示偏振度显著图的平均值;${\sigma _{Dop}}$ 表示偏振度显著图的标准差;$k$ 为可调节的常数参数。当${C_{Dop}}$ 值大于阈值$T{h_{Dop}}$ 时,该像素为目标,反之则为背景。 -
文中数据采集所使用的分焦平面红外偏振相机是将检偏器件通过微纳加工覆盖在焦平面上,焦平面上每个像元对应一个微偏振片,由微纳光栅构成的微偏振片阵列中每2×2个像元为一组。
如图1所示,图1的左侧为2×2的微偏振片阵列,对应的偏振角度分别为:左上角0°、右上角45°、右下角90°、左下角135°,图1右侧为由2×2的微偏振片阵列重复排列所构成的红外偏振马赛克图像。
基于图1的红外偏振马赛克图像,文中利用偏振梯度直方图(Polarization Gradient Histogram)[20]作为特征提取法,跟踪算法采用KCF[21]跟踪算法。
-
基于FPGA的场景一的车辆目标检测跟踪结果如图8所示,图8(a)~(d)为第1、20、45、60帧的目标检测结果,图(e)为Dop图像,图(f)为Dop显著图像。场景一在跟踪过程中存在相似的车辆目标干扰,可以发现道路上的车辆在偏振度图像上较为明显,因此跟踪中并未出现错误跟踪的情况。
基于FPGA的场景二的车辆目标检测跟踪结果如图9所示,图9(a)~(d)为第1、20、60、80帧的目标检测结果,图(e)为Dop图像,图(f)为Dop显著图像。场景二在跟踪过程中存在目标被遮挡的干扰,从偏振显著图可以看到目标的偏振显著度非常高,因此跟踪始终保持着非常高的准确性。
基于FPGA的场景三的车辆目标检测跟踪结果如图10所示,图10(a)~(d)为第1、20、60、80帧的目标检测结果,图(e)为Dop图像,图(f)为Dop显著图像。场景三中目标的红外辐射强度与背景相近,甚至低于部分背景。在跟踪过程中,由于文中的算法引入了偏振信息,因此具有较高的跟踪精度,解决了目标红外辐射强度较弱下的目标跟踪问题。
基于FPGA的场景四的飞机目标检测跟踪结果如图11所示,图11(a)~(d)为第1、20、60、90帧的目标检测结果,图(e)为Dop图像,图(f)为Dop显著图像。场景四中飞机目标的红外辐射强度与背景相近,利用红外辐射强度信息难以实现检测跟踪。由偏振度与偏振显著度图可以看到飞机的偏振信息与背景差异较大,因此文中的算法具有较高的检测跟踪精度,解决了目标红外辐射强度较弱下的目标检测跟踪问题。
对检测跟踪的实时性进行的测试结果表明,文中的检测跟踪系统的处理帧率能够达到25 fps,满足实时性的要求。
-
系统的功耗和硬件资源消耗情况关系到该硬件平台的使用场景,低功耗的硬件系统可以在特殊场景下使用,例如在机载环境、野外作业环境等。因此该小节对硬件系统的功耗以及资源使用率进行测试。
在Vivado软件中可以对系统的资源使用情况和功耗进行查看,见表1。从表中可以看出,该系统对硬件资源消耗较少,所消耗的资源比率较低。
表 1 系统硬件资源消耗情况
Table 1. System hardware resource consumption
Resource Used Usabl LUT 15440 171900 LUTRAM 665 70400 FF 18852 343800 BRAM 14 500 DSP 29 900 IO 8 250 MMCM 1 8 因此,该系统所使用的硬件资源只占了该硬件平台的一小部分,为低功耗的要求奠定了基础。硬件资源的消耗利用率如图12所示,各个硬件资源的利用率即均没有超过资源总量的15%,为低功耗的要求奠定了基础。
该系统的功耗情况如图13所示。系统的功耗分为静态功耗和动态功耗,总功耗为2.255 W,其中静态功耗为0.223 W,动态功耗为2.032 W。动态功耗中以PS部分功耗占比最多,为75%。因此可知,该系统的功耗情况满足低功耗要求,适用于一些特殊场景环境。
Maneuvering object detection and tracking system based on infrared polarization imaging (Invited)
-
摘要: 基于红外偏振摄像的机动目标检测跟踪系统对实时性要求较高,而且在野外场景下需要具备低功耗的特点。FPGA具有并行计算的特性,能够极大的提高系统吞吐量和处理数据速度,能够满足实时性的要求,因此一种基于FPGA的目标检测跟踪系统被设计出来并得以实现。在硬件开发平台上采用模块化以及软硬件协同设计的方式,将具有不同计算特点的任务分别在PS端(ARM)以及PL端(FPGA)实现,其中PL部分负责部分算法的加速、FPGA和ARM处理器之间数据传输以及HDMI等接口逻辑控制等,PS部分负责实现较为复杂的检测跟踪算法,以及负责控制FPGA端的各个模块。最后,对目标检测跟踪系统进行实验测试和分析,给出系统的硬件资源消耗及功耗,结果表明该目标检测跟踪系统能够满足实时性的要求,并且具备低功耗的特点。Abstract: The infrared polarization object tracking under dynamic scenes has a demand for real-time performance and low power consumption. FPGA has the characteristics of parallel computing, which can greatly improve the system throughput and data processing speed, and can meet the requirements of real-time. Therefore, a target detection and tracking system based on FPGA was designed and implemented. On the hardware development platform, the methods of modularization and software and hardware collaborative design were adopted to realize the tasks with different computing characteristics in PS (ARM) and PL (FPGA). PL was responsible for the acceleration of some algorithms, data transmission between FPGA and ARM processors, HDMI and other interface logic control. PS was responsible for the implementation of more complex detection and tracking algorithms, and controlled each module in FPGA. Finally, the target detection and tracking system was tested and analyzed, and the hardware resource consumption and power consumption of the system were also given in the experiment. The results showed that the purposed target detection and tracking system can meet the requirements of real-time and low power consumption..
-
Key words:
- infrared polarization /
- object tracking /
- object detection /
- FPGA
-
表 1 系统硬件资源消耗情况
Table 1. System hardware resource consumption
Resource Used Usabl LUT 15440 171900 LUTRAM 665 70400 FF 18852 343800 BRAM 14 500 DSP 29 900 IO 8 250 MMCM 1 8 -
[1] Li Bo. Application and development trend of infrared stealth technology [J]. Chinese Optics, 2013, 6(6): 818-823. [2] Liu Yi, Shi Haodong, Jiang Huilin, et al. Infrared polarization properties of targets with rough surface [J]. Chinese Optics, 2020, 13(3): 459-471. [3] Zhang Yan, Han Jiantao, Li Jicheng, et al. Characteristics analysis of infrared polarization for several typical artificial objects[C]//Image and Signal Processing for Remote Sensing. International Society for Optics and Photonics, 2014: 92440I. [4] Luo Haibo, Zhang Junchao, Gai Xingqin, et al. Development status and prospects of polarization imaging technology (Invited) [J]. Infrared and Laser Engineering, 2022, 51(1): 20210987. (in Chinese) [5] Forssell Goran. Test and analysis of the detectability of personnel mines in a realistic minefield by polarization in the infrared LW region[C]//Defense and Security. International Society for Optics and Photonics, 2004: 187-195. [6] Forssell Goran. Passive IR polarization measurements applied to covered surface landmines[C]//International Society for Optics and Photonics, 2003: 547-557. [7] Cremer Frank, Wim De Jong, Klamer Schutte. Infrared polarization measurements and modelling applied to surface laid anti-personell landmines [J]. Optical Engineering, 2002, 41(5): 1021-1032. doi: 10.1117/1.1467362 [8] Aron Yoram, Gronau Yuval. Polarization in the LWIR: A method to improve target acquisition [C]//Proceedings of SPIE, 2005, 5783(1): 653-661. [9] Bradley Ratliff, Daniel LeMaster, Robert Mack, et al. Detection and tracking of RC model aircraft in LWIR microgrid polarimeter data[C]//Proceedings of SPIE, 2011, 8160: 25-31. [10] Zhao Yongqiang, Dai Huimin, Shen Linghao, et al. Review of underwater polarization clear imaging methods [J]. Infrared and Laser Engineering, 2020, 49(6): 20190574. (in Chinese) doi: 10.3788/irla.4_2019-0574 [11] Gong Jian, Liu Liang, Qiu Rongchao, et al. Adaptive scale local contrast detection for infrared polarization ship targets [J]. Optics and Precision Engineering, 2020, 28(1): 223-233. (in Chinese) doi: 10.3788/OPE.20202801.0223 [12] Meng Lu, Yang Xu. A survey of object tracking algorithms [J]. Acta Automatica Sinica, 2019, 45(7): 1244-1260. (in Chinese) [13] Lu Ruitao, Yang Xiaogang, Shen Lurong, et al. Robust distribution fields tracking based on bayesian mutual information[C]//2018 Chinese Automation Congress, 2018: 919-925. [14] Yuan Yuan, Xiong Zhitong, Wang Qi. An incremental framework for video-based traffic sign detection, tracking, and recognition [J]. IEEE Transactions on Intelligent Transportation Systems, 2017, 18(7): 1918-1929. doi: 10.1109/TITS.2016.2614548 [15] Wan Minjie, Gu Guohua, Qian Weixian, et al. Unmanned aerial vehicle video-based target tracking algorithm using sparse representation [J]. IEEE Internet of Things Journal, 2019, 6(6): 9689-9706. doi: 10.1109/JIOT.2019.2930656 [16] Sam Hare, Stuart Golodetz, Amir Saffari, et al. Struck: structured output tracking with kernels [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(10): 2096-2109. doi: 10.1109/TPAMI.2015.2509974 [17] Hua Weixin, Mu Dejun, Guo Dawei, et al. Visual target tracking based on multi-view feature fusion with online multiple instance learning[C]//2017 36 th Chinese Control Conference (CCC), 2017: 10783-10788. [18] Guo Qing, Feng Wei, Zhou Ce, et al. Structure-regularized compressive tracking with online data-driven sampling [J]. IEEE Transactions on Image Processing, 2017, 26(12): 5692-5705. doi: 10.1109/TIP.2017.2745205 [19] Han Jinhui, Ma Yong, Zhou Bo, et al. A robust infrared small target detection algorithm based on human visual system [J]. IEEE Geoscience and Remote Sensing Letters, 2014, 11(12): 2168-2172. doi: 10.1109/LGRS.2014.2323236 [20] Qiao Xinbo, Chen Lulu, Zhao Yongqiang. Polarization gradient histogram for object tracking in infrared polarization imaging: A feature extraction method for polarization mosaic image[C]//2021 10th International Conference on Computing and Pattern Recognition, 2021: 224-230. [21] Joao Henriques, Rui Caseiro, Pedro Martins, et al. High-speed tracking with kernelized correlation filters [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(3): 583-596. doi: 10.1109/TPAMI.2014.2345390