Volume 52 Issue 3
Mar.  2023
Turn off MathJax
Article Contents

Li Ronghua, Wang Meng, Zhou Wei, Fu Jiaru. Pose estimation of flying target based on bi-modal information fusion[J]. Infrared and Laser Engineering, 2023, 52(3): 20220618. doi: 10.3788/IRLA20220618
Citation: Li Ronghua, Wang Meng, Zhou Wei, Fu Jiaru. Pose estimation of flying target based on bi-modal information fusion[J]. Infrared and Laser Engineering, 2023, 52(3): 20220618. doi: 10.3788/IRLA20220618

Pose estimation of flying target based on bi-modal information fusion

doi: 10.3788/IRLA20220618
Funds:  Young Scientists Fund of the National Natural Science Foundation of China (41901415); Science and Technology Foundation of State Key Laboratory Grant Number (2022-JCJQ-L8-015-0201); The Key Project of Scientific Research Project of Education Department of Liaoning Province (LJKZ0475); Dalian High-Level Talent Innovation Support Program Grant Number (2022RJ03)
  • Received Date: 2022-08-10
  • Rev Recd Date: 2022-09-15
  • Publish Date: 2023-03-25
  •   Objective   Flying target pose estimation is the key technology to realize trajectory prediction and missile guidance control. Real-time calculation of missile posture is conducive to judge whether the missile hits the target, timely detect missile failure, and carry out early destruction. The development of information and intelligent technology has led to the improvement of data acquisition accuracy of color camera, laser radar and other sensors, which has formed a technical system of data acquisition by sensors and target position and attitude estimation by relevant algorithms. Most of the existing target pose estimation methods can effectively detect and estimate the target pose. However, there are some problems in the precision prediction and guidance control of missile landing point, such as the inability to quickly and accurately extract and estimate the position and attitude of the flying target in the complex background. Therefore, on the premise of ensuring being real-time, a method for estimating the pose of flying targets based on area array lidar and bi-modal information fusion is proposed.  Methods   Firstly, the coordinate transformation model between the camera and the lidar is established to realize the pixel level matching of the two sensors, fusing the image and point cloud at the same time (Fig.2); Secondly, the ViBe (Visual Background Extractor) algorithm and depth information fusion algorithm are used to extract the moving target in the image, and select the corresponding point cloud according to the image moving target position box (Fig.5); Finally, the PnP (Perspective-n-Point) algorithm is used for rough registration of feature points (Fig.8), to obtain the initial rotation and translation matrix between point clouds. And using I-Kd Tree (Incremental K-dimensional Tree) to accelerate the search of adjacent points, the ICP (Iterative Close Point) algorithm is used for accurate registration, to improve the registration speed.  Results and Discussions   Simulation test and hardware-object simulation test are used to verify the accuracy and stability of the method. The results show that the accuracy of the two-dimensional image object detection algorithm is 97% (Tab.3), and the error classification ratio is 0.0112% (Tab.3). Compared with the traditional ICP algorithm, the accuracy of the pose estimation algorithm is improved by 53.2% (Tab.2), the single time consumption is reduced to 132 ms from 261 ms (Tab.2). Compared with other algorithms, the pose estimation algorithm also has certain advantages.  Conclusions   An algorithm for estimating the pose of flying targets based on bi-modal information fusion is proposed, which can effectively estimate the pose of flying targets on the basis of selecting appropriate parameters. The accuracy of the algorithm is verified by simulation tests. 50 frames of data are simulated and the average error is calculated under the initial condition that the initial object distance between the target and the lidar is 30 m. The simulation results show that the X-axis error is 1.06 mm, the Y-axis error is 4.59 mm, the Z-axis error is 2.07 mm, the Y-axis rotation angle error is 0.63°, the Z-axis rotation angle error is 1.01°, and the solution time is 132 ms. The accuracy of the algorithm is verified by semi-physical ground experiments, and the precision (P) in the image target extraction test is 0.97, the recall (R) is 0.844, and the percentage of wrong classification (PWC) is 0.011 2%. The statistical average error in the pose estimation test is respectively 4.9 mm for X-axis error, 2.7 mm for Y-axis error, 4.62 mm for Z-axis error, 0.97° for Y-axis rotation angle error and 0.89° for Z-axis rotation angle error. The defect that the single source data of the proposed method is difficult to describe the moving target comprehensively can be remedied, and an objective solution is provided for the position and attitude estimation of the flying target. The proposed method is applied to the accurate prediction and guidance control of the landing point of flying targets, and has high military application value.
  • [1] Pan Chang. An improved algorithm of missile falling point prediction [J]. Journal of Ordnance Equipment Engineering, 2018, 39(9): 30-32+46. (in Chinese)
    [2] Liu Junliang, Chen Shangfeng, Lu Huanzhang, et al. Research on infrared irradiance data stimulation for ballistic targets recognition [J]. Infrared and Laser Engineering, 2016, 45(10): 39-45. (in Chinese)
    [3] Li Jiawei, Jiang Jing, Liu Chongyang, et al. Modeling and simulation for target complex trajectory of ballistic missile [J]. Journal of System Simulation, 2020, 32(8): 1515-1523. (in Chinese)
    [4] Shao Huibing, Cui Naigang, Wei Changzhu. Multi-constrained intelligent trajectory planning for gliding missiles [J]. Guangxue Jingmi Gongcheng/Optics and Precision Engineering, 2019, 27(2): 410-420. (in Chinese)
    [5] Le Ying, Zhao Zhicheng. Multi-moving object detection and segmentation based on background difference method [J]. Chinese Journal of Construction Machinery, 2020, 18(4): 305-309. (in Chinese)
    [6] Barnich O, van Droogenbroeck M. ViBe: a universal background subtraction algorithm for video sequences [J]. IEEE Transactions on Image Processing, 2011, 20(6): 1709-1724. doi:  10.1109/TIP.2010.2101613
    [7] Liu Jiajun, Lin Haokun. Adaptive threshold moving target detection algorithm based on ViBe method [J]. Laser & Optoelectronics Progress, 2022, 59(2): 144-153. (in Chinese)
    [8] Hu Tao, Zhu Xinyan, Guo Wei, et al. A moving object detection method combining color and depth data [J]. Geomatics and Information Science of Wuhan University, 2019, 44(2): 276-282. (in Chinese)
    [9] Yin Fang, Wu Yun. Model reconstruction and pose optimization of non-cooperative rotating space target [J]. Guangxue Jingmi Gongcheng/Optics and Precision Engineering, 2019, 27(8): 1854-1862. (in Chinese)
    [10] Tian Guangbao, Wang Jian, Wang Bowen. Monocular camera non-cooperative target extraction and pose detection [J]. Infrared and Laser Engineering, 2021, 50(12): 536-544. (in Chinese)
    [11] Liu yaohua, Ma yue, Xu min. Point cloud instance segmentation based robust multi-target [J]. Laser & Optoelectronics Progress, 2022, 20(8): 1-18. (in Chinese)
    [12] Kechagias Stamatis O, Aouf N. Dubanchet V, Richardson M. A. DeepLO: Multi-projection deep LIDAR odometry for space orbital robotics rendezvous relative navigation [J]. ActaAstronautica, 2020, 177: 270-285. doi:  10.1016/j.actaastro.2020.07.034
    [13] 谭长生, 黄庚华, 王凤香, 等. 空间位姿测量中面阵点云CPD方法的优化及验证[J]. 红外与激光工程: 1-12[2022-09-19].

    Tan Changsheng, Huang Genghua, Wang Fengxiang et al. Optimization and Validation of Coherent Point Drift for Planar-array- Based Point Cloud in Space Pose Measuremn[J]. Infrared and Laser Engineering, 1-12[2022-09-19]. (in Chinese)
    [14] Bi Yong, Pan Mingqi, Zhang Shuo, et al. Overview of 3 D point cloud super-resolution technology [J]. Chinese Optics, 2022, 15(2): 210-223. (in Chinese)
    [15] Li Ronghua, Wen Shuaifang, Xiao Yuzhi, et al. Indexes optimization macro model of on-orbit imaging lidar in space [J]. Infrared and Laser Engineering, 2018, 280(2): 164-171. (in Chinese)
    [16] Lv Yuanzhi, Sun Qiang, Bi Guoling. Recognition and repairing of surface hole in three dimensional laser scanning system [J]. Chinese Optics, 2016, 9(1): 114-121. (in Chinese) doi:  10.3788/co.20160901.0114
    [17] Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing [J]. arXiv preprint arXiv: 2107.13586, 2021.
    [18] Li Ronghua, Xue Haopeng, Yang Youqing, et al. Linear array lidar imaging mechanism and 3 D reconstruction for space instable targets [J]. Journal of Astronautics, 2020, 41(8): 1049-1057. (in Chinese)
    [19] Xu Fei, Xu Shimeng, Du Jianhua, et al. Research on trajectory correction of equipment with autonomous precise fixed-point drop [J]. Journal of Ordnance Equipment Engineering, 2017, 38(6): 165-168. (in Chinese)
    [20] Fang Yi, Zhang Xiuda, Hu Jian, et al. A method applying gray image superposition to improve ranging accuracy in planar array laser radar [J]. Chinese Journal of Lasers, 2013, 40(1): 0114002-230. (in Chinese) doi:  10.3788/CJL201340.0114002
    [21] Cai Yue, Xu Wenbing, Liang Dan, et al. Impacts of different factors on accuracy of point cloud obtained from terrestrial three-dimensional laser scanning [J]. Laser & Optoelectronics Progress, 2017, 54(9): 092801-373. (in Chinese)
  • 加载中
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

Figures(18)  / Tables(11)

Article Metrics

Article views(74) PDF downloads(36) Cited by()

Related
Proportional views

Pose estimation of flying target based on bi-modal information fusion

doi: 10.3788/IRLA20220618
  • 1. Institute of Mechanical Engineering, Dalian Jiaotong University, Dalian 116028, China
  • 2. No.91550 Unit of the PLA, Dalian 116023, China
Fund Project:  Young Scientists Fund of the National Natural Science Foundation of China (41901415); Science and Technology Foundation of State Key Laboratory Grant Number (2022-JCJQ-L8-015-0201); The Key Project of Scientific Research Project of Education Department of Liaoning Province (LJKZ0475); Dalian High-Level Talent Innovation Support Program Grant Number (2022RJ03)

Abstract:   Objective   Flying target pose estimation is the key technology to realize trajectory prediction and missile guidance control. Real-time calculation of missile posture is conducive to judge whether the missile hits the target, timely detect missile failure, and carry out early destruction. The development of information and intelligent technology has led to the improvement of data acquisition accuracy of color camera, laser radar and other sensors, which has formed a technical system of data acquisition by sensors and target position and attitude estimation by relevant algorithms. Most of the existing target pose estimation methods can effectively detect and estimate the target pose. However, there are some problems in the precision prediction and guidance control of missile landing point, such as the inability to quickly and accurately extract and estimate the position and attitude of the flying target in the complex background. Therefore, on the premise of ensuring being real-time, a method for estimating the pose of flying targets based on area array lidar and bi-modal information fusion is proposed.  Methods   Firstly, the coordinate transformation model between the camera and the lidar is established to realize the pixel level matching of the two sensors, fusing the image and point cloud at the same time (Fig.2); Secondly, the ViBe (Visual Background Extractor) algorithm and depth information fusion algorithm are used to extract the moving target in the image, and select the corresponding point cloud according to the image moving target position box (Fig.5); Finally, the PnP (Perspective-n-Point) algorithm is used for rough registration of feature points (Fig.8), to obtain the initial rotation and translation matrix between point clouds. And using I-Kd Tree (Incremental K-dimensional Tree) to accelerate the search of adjacent points, the ICP (Iterative Close Point) algorithm is used for accurate registration, to improve the registration speed.  Results and Discussions   Simulation test and hardware-object simulation test are used to verify the accuracy and stability of the method. The results show that the accuracy of the two-dimensional image object detection algorithm is 97% (Tab.3), and the error classification ratio is 0.0112% (Tab.3). Compared with the traditional ICP algorithm, the accuracy of the pose estimation algorithm is improved by 53.2% (Tab.2), the single time consumption is reduced to 132 ms from 261 ms (Tab.2). Compared with other algorithms, the pose estimation algorithm also has certain advantages.  Conclusions   An algorithm for estimating the pose of flying targets based on bi-modal information fusion is proposed, which can effectively estimate the pose of flying targets on the basis of selecting appropriate parameters. The accuracy of the algorithm is verified by simulation tests. 50 frames of data are simulated and the average error is calculated under the initial condition that the initial object distance between the target and the lidar is 30 m. The simulation results show that the X-axis error is 1.06 mm, the Y-axis error is 4.59 mm, the Z-axis error is 2.07 mm, the Y-axis rotation angle error is 0.63°, the Z-axis rotation angle error is 1.01°, and the solution time is 132 ms. The accuracy of the algorithm is verified by semi-physical ground experiments, and the precision (P) in the image target extraction test is 0.97, the recall (R) is 0.844, and the percentage of wrong classification (PWC) is 0.011 2%. The statistical average error in the pose estimation test is respectively 4.9 mm for X-axis error, 2.7 mm for Y-axis error, 4.62 mm for Z-axis error, 0.97° for Y-axis rotation angle error and 0.89° for Z-axis rotation angle error. The defect that the single source data of the proposed method is difficult to describe the moving target comprehensively can be remedied, and an objective solution is provided for the position and attitude estimation of the flying target. The proposed method is applied to the accurate prediction and guidance control of the landing point of flying targets, and has high military application value.

    • 随着信息化、智能化技术的不断发展,现代战争对武器装备打击的精准度要求越来越高。导弹飞行试验中实时解算导弹位姿、准确预报导弹落点有利于判断导弹是否击中目标,并且可以及时发现导弹故障实施提前炸毁[1]。与传统的建立动力学、运动学数学模型推测导弹落点[2]不同,如今通过对飞行过程中导弹位姿信息进行捕捉,并根据抛物线近似、非线性滤波、深度学习等方法预测导弹落点[3-4],符合信息化、智能化的要求。因此,实现对复杂背景下的飞行目标实现快速位姿估计,成为了目前亟需解决和攻克的问题。

      国内外学者针对复杂背景下飞行目标检测、位姿估计问题提出大量研究方法。例如,乐英等[5]等提出了基于背景差分和帧间差分法对目标进行检测,增强了背景差分法的检测能力,使提取的运动目标相对完整且噪声少。Barnic等[6]提出ViBe (Visual Background Extractor) 算法对运动目标进行检测,利用图像初始帧来构建背景模型,计算量小且实时性高,但是当初始帧包含运动目标时,容易将运动目标初始化到背景模型中产生伪前景。刘家军等[7]针对ViBe算法检测运动目标时产生伪前景的问题,在Vibe算法的基础上使用熵值法判断颜色和空间位置在相似程度函数中的权重,采用迭代法确定自适应阈值,以增强前景分割精度,但ViBe算法容易受光照影响,在面临光照变化、动态干扰等情况时检测性能较差。胡涛[8]等提出RGBD相机与ViBe算法结合,将深度信息作为背景建模的样本,提高了ViBe算法的稳定性,但该算法无法直接应用于彩色相机分辨率和雷达分辨率不同的情况。尹芳[9]采用ICP算法计算相邻关键帧之间的相对位姿信息,但对于飞行速度快的目标,ICP配准算法无法处理初始位置相差较大的点云并易陷入局部最优。田光宝等[10]采用3个不共线的特征点对目标进行位姿求解,但在特征点提取时容易产生较大误差影响位姿估计精度;刘耀华等[11]利用聚类算法和随机森林算法对点云进行分割,然后对分割后的实例采用FLANN算法进行特征匹配得到实例分割后特征点所对应的匹配点。最后通过RANSAC算法和ICP算法求解目标的精确位姿,但该算法不适用于背景复杂且目标物小的场景。Kechagias [12]提出了激光雷达和相机融合的方法估计目标位姿,利用点云与RNN网络结合进行目标相对姿态估计。但飞行目标视觉特征不明显,在训练过程中卷积神经网络容易被忽略,且目标在飞行过程中存在尺度变化幅度大的情况,也给训练带来了很大困难。尽管上述方法可以有效对目标进行检测和位姿估计,但在导弹落点精准预测和制导控制上存在无法在复杂背景下对飞行目标快速、准确提取以及位姿估计的问题。

      三维测量中线阵雷达主要依赖线列推扫式,其优势在于灵活的视场配置和较高的点云密度,但受限于扫描速率,在对动态目标进行观测时,其测量结果与实际轮廓存在较大差异。而基于高灵敏度阵列的面阵激光雷达,可实现单脉冲瞬时成像,有效解决了运动扫描导致的时间同步问题和条带错位现象[13]

      因此,在保证实时性的前提下,文中提出基于面阵式激光雷达与图像信息融合的双模态飞行目标位姿估计方法。首先,采用ViBe算法与深度信息结合,实现图像目标快速提取并且可以较好的抑制噪点;其次,通过将整体点云和图像目标融合,实现从整体点云中提取出目标点云;最后,采用PnP (Pespective-n-Point) 算法获取点云间初始旋转平移矩阵并采用基于IK-D Tree (Incremental K-dimensional Tree) 的ICP (Iterative Close Point) 算法解算精确位姿。文中方法与现有方法相比计算速度快、提取目标精度高,能够很好应对背景复杂的场景特殊性。

    • 文中所提方法的总体框架包含两个部分:图像目标提取部分和点云信息处理部分,其中点云信息处理包含特征叠加和目标点云提取,其流程如图1所示。

      Figure 1.  Flow chart of bimodal fusion algorithm

    • 研究中所用彩色相机的出厂基本参数不能满足高精度估计要求,且彩色相机在成像过程中会因镜头制造误差和组装误差等原因导致成像效果不理想。因此,采用张正友相机标定法对彩色相机内参进行重新标定。标定完成后采用PnP算法求解激光雷达至彩色相机的映射矩阵。具体方法为:用彩色相机和激光雷达对同一特征目标进行拍摄,采用手动标记的方式标记出特征点在彩色相机和点云中的位置。假设标记的特征点在激光雷达坐标系下的坐标为(XL,YL,ZL),在图像像素坐标系下的坐标为(u, v)可建立如下关系:

      式中:Q表示彩色相机内参;RLC表示激光雷达坐标系到彩色相机坐标系的旋转矩阵;TLC表示平移矢量;m11~m34表示待求矩阵的12个元素。

      通过提取多组激光雷达与图像对应特征点,建立如下约束方程组,即可求解出矩阵的12个元素。由公式(1)展开可得到:

      求解公式(2)至少需要6组特征点,为减小误差采集多组特征点进行解算。应用最小二乘法求得映射矩阵初始解,并用Levenberg-Marquarat算法对其结果进行优化得到最终结果。

    • 求解出彩色相机内参和激光雷达至彩色相机的映射矩阵后,通过转换矩阵实现点云坐标到图像像素坐标的投影。为了实现点云坐标与图像像素坐标的相互转换,将各变换矩阵扩展为4×4的方阵。若将点云坐标表示为P(XL,YL,ZL),映射到图像像素坐标系的二维映射点云表示为Y(u ,v),则点云坐标到图像像素坐标的映射计算方式如下:

      式中:M0为扩充后的外参矩阵。

      将点云数据转换至图像像素坐标系,即可完成点云数据与图像数据的特征叠加。点云与图像融合效果如图2所示。

      Figure 2.  Point cloud and image fusion

    • 针对在传统ViBe算法中存在的因光照强度改变而影响算法精度的问题,利用激光雷达不受光照影响的优势,采用ViBe算法与深度信息融合,将深度信息加入背景模型样本中作为运动目标检测的权值。

    • 文中采用将点云信息通过坐标转换投影至图像上的方式赋予每个像素点深度值。但受到当前传感器芯片技术水平的限制,直接采集到的三维点云数据通常分辨率较低,远不及可见光图像[14],无法做到给每个像素赋予深度信息。因此需要对点云进行稠密化。

      由面阵式激光雷达扫描成像原理可知,点云在空间的分布情况为呈锥形方向均匀展开。因此通过视场角对相同物距的某一平面激光雷达可视区域的4个角点计算,即可得出雷达在此平面的最大投影面积[15]。假设雷达到测量面的距离为L,彩色相机分辨率为:aC×bC,雷达分辨率为aL×bL,雷达水平视场角为H,垂直视场角为G,4个角点的分别为P1P2P3P4,4个角点的坐标为:${{{P}}_{{1}}}{{( - L}} \cdot {\tan} \left(\dfrac{{{H}}}{{{2}}}\right),{L} \cdot {\tan}\left(\dfrac{{{G}}}{{{2}}}\right),{L)}$${{{P}}_{{2}}}{{(L}} \cdot {{\tan}}\left( \dfrac{{{H}}}{{{2}}} \right),{{L}} \cdot {{\tan}}\left( \dfrac{{{G}}}{{{2}}} \right),{{L)}}$${{{P}}_{{3}}}{{( - L}} \cdot {{\tan}}\left( \dfrac{{{H}}}{{{2}}} \right), {{- L}} \cdot {{\tan}}\left( \dfrac{{{G}}}{{{2}}} \right), {{L)}}$$ P\left(L-\tan \left(\dfrac{H}{2}\right),-L-\tan \left(\dfrac{G}{2}\right), L\right) $。假设P1P2P3P4对应像素坐标分别为:Y1(u1, v1)、Y2(u2, v2)、Y3(u3, v3)、Y4(u4,v4),则点云转化至图像坐标系后,投影最大面积为$ 4\cdot {u}_{2}\cdot {v}_{2} $,深度信息在彩色图像上的占比率V$ \dfrac{4\cdot {u}_{2}\cdot {v}_{2}}{{a}_{L}\cdot {b}_{L}} $

      在实际应用中由于遮挡物(地面、天花板)等原因会导致映射至图像坐标系后,某一部分区域深度信息在彩色图像上的占比率会大于V,图像中一个像素点周围有$ {\left(2 r+1\right)}^{2}-1 $个像素(其中r为最远处像素距中心像素的水平距离),因此将不小于V的最小$\, {\left(2 r+1\right)}^{2}-1 $作为深度信息填充的阈值,填充效果如图3所示。

      Figure 3.  Point cloud densification

      填充方法如下:

      (1)建立零矩阵M(aC·bC), 将点云投影至彩色相机坐标系后对坐标信息进行向下取整,根据坐标将深度值填充至M中。

      (2)寻找出M中的每个零点坐标,在周围距离为r的像素内寻找非零点,将所有非零点的均值作为该点的深度值。

      在稠密深度信息构建完成后,使用三张背景图作为背景模型样本集合。在三张背景图的每一像素的8个相关邻域中随机选取5个像素值组成该坐标的像素背景模型;然后,对三张背景图对应的点云信息进行1.3.1节中所说的融合、稠密化处理,并选取对应坐标的深度值作为该坐标的深度背景模型。

    • 背景建模完成后,需要根据阈值区分前景和背景。文中采用参考文献[6]所提算法获取像素确定自适应阈值,采用最大类间方差法获取深度信息确定自适应阈值。

      首先使用背景深度矩阵减去含运动目标的深度矩阵并向下取整,记为深度差值矩阵U,并查找矩阵U中最大的深度差值,记为L。然后计算矩阵U中深度值为i的点在矩阵中的分布概率Pi,其中:$ {P}_{\text{i}}= \dfrac{U中深度差值为{i}的点数}{U总点数} $,( i=1,2,3·····L)。则最大类间方差构建方法为:

      式中:Pi为矩阵G中每个深度值在矩阵中的分布概率,(i=1,2,3·····L);k为所求自适应阈值;m为灰度级k的累加均值;mG为图像全局均值。

      将公式(5)~(7)代入公式(8)中,求解出令公式(8)最大化的差值k即为最大类间方差法的阈值。

    • 在实际应用中,由于扫描设备自身的限制或待测地形(物)的特点,获取的点云数据中往往会有大量的空洞[16]。为避免空洞点对整体算法的影响,当某一像素点对应的深度背景模型深度值为0时仅使用像素值作为运动目标检测的权值。在图像目标提取完成时采用中值滤波去除图像中的离群噪点,提取效果如图4所示。

      Figure 4.  Moving object extraction

      由于目标表面像素值与背景像素值接近、目标表面由于镜面反射成像后颜色信息发生改变等原因,导致提取出的目标有缺陷、不完整。因此使用图像形态学中的腐蚀和膨胀操作对目标缺陷部分进行填充。先采用膨胀操作对二值化物体进行扩充,再采用腐蚀操作对图像大小进行还原即可得到完整的运动目标,效果如图5所示。

      Figure 5.  Image morphological processing

    • 对1.3节中求解到的目标图像使用“sobel”算子提取目标边缘轮廓的像素坐标,并将点云坐标投影至图像像素坐标。通过比对目标区域和投影点云坐标去判断出目标投影点云的区域。

      在点云投影至图像像素坐标系后,点云的相对位置并未改变,因此通过观察目标投影点云在整体投影点云的位置即可得到目标三维点云在原始点云中的位置,效果如图6(a)所示。在框选目标投影点云范围选择点云时在边缘部分容易提取到背景的点云信息,因此最后对提取出的点云进行去噪处理即可得到目标的三维点云,效果如图6(b)所示。

      Figure 6.  Target point cloud extraction

    • 使用ICP算法对飞行目标进行位姿估计时,存在以下两个问题:

      (1) 由于提取到的点云目标与点云模型之间距离相差大。传统ICP算法在两个待配点云欧式距离较远时,会造成匹配错误,影响最终匹配效果;

      (2) 传统ICP算法进行最近点查找时需计算欧氏距离最短的点。查找速度慢,配准效率低。

      针对该问题,采用基于图像和点云特征点关系的PnP算法获取点云间初始旋转平移矩阵,完成点云目标与点云模型粗配准。采用IK-D Tree优化加速临近点搜索。

    • 对1.3节中得到的运动区域采用LSD直线检测算法检测直线段,然后采用霍夫变换对直线进行提取。在霍夫变换时存在一条直线段被提取成多条直线段的情况。因此,根据直线段到坐标原点的距离和直线段与坐标原点的夹角将直线段划分为4类,将划分在同一组的直线采用最小二乘法进行直线拟合,从而得到所需直线,特征直线提取如图7所示。最后,通过计算直线间的交点得到4个目标特征点的图像像素坐标。

      Figure 7.  Feature line extraction

      根据目标真实尺寸采用相关软件进行目标三维点云模型创建。以模型几何中心作为坐标原点建立物体坐标系。采用手动标点的方式得到点云模型的4个特征点。待测目标物体模型及其坐标系定义如图8所示。已知4个特征点在图像像素坐标系和在物体坐标系下的坐标,利用PnP算法计算出目标在图像像素坐标系下的位姿信息,粗配准效果如图9所示。

      Figure 8.  Point cloud model coordinate system

      Figure 9.  Pose solution results

      假设求得的旋转矩阵和平移矩阵分别为:Rcp(3×3)和Tcp(1×3),目标原始点云在物体坐标系下的坐标为P(XM,YM,ZM),目标在图像像素坐标系下的点云为PFt(XF,YF,ZF)。对各变换矩阵全部扩展成4×4的方阵,则计算公式为:

      最后,利用公式(4)将点云PFt从图像像素坐标系转换至激光雷达坐标系,即完成点云粗配准。

    • 采用PnP算法获得粗略六维位姿后,模型点云与目标点云已大致重合,但精度较低。为提高配准精度,采用基于IK-D Tree[17]的ICP算法实现点云精配准。IK-D Tree加速对近邻点搜索构建思路如下:

      (1) 按照模型点云中点集的最大协方差构建传统K-D Tree,并用K近邻搜索(k Nearest Neighbor, kNN)查询点距离最近点;

      (2)对未匹配的子树采用Lazy LabelInit进行删除,并使用pullup函数将未匹配的子树信息储存到节点Ti(i=1,2,3···n)。其中包括treesize (子树的尺寸);invalidnum (子树上被标记为“deleted”的节点数字),以及range (以当前节点为根节点的子树中,所有节点在三维空间中的最小包络盒),完成IK-D Tree构建。

      (3)检测IK-D Tree的平衡性。平衡准则由两个子准则组成:α-平衡准则和α-删除标准。假设IK-D tree的一个子树的根在Tn。当子树满足以下条件:

      式中:α∈(0.5,1);S(Tn)是节点Tn的treesize属性;I(Tn)表示子树上无效节点数量。

      αbalαdel定义为:

      如果IK-D Tree不满足任一条件,则需对IK-D Tree进行双线程重建。

      (4) 在完成IK-D Tree构建后利用节点的 “range”信息加速临近点搜索。

    • 查找到最近点后,采用奇异值分解算法求取位姿增量,获取其相对坐标转换关系:旋转矩阵Rt与平移矩阵Tt。通过迭代的方式求解使得公式中目标函数达到最小的刚体变换,并将该变换作用于源点云,迭代进行这一过程直到满足某一设定的收敛准则。

      式中:aibi分别代表两组扫描数据中各点的三维坐标。当配准结果相对于真实值三轴位姿精度小于1°时迭代终止。

      通过以上配准过程获得最优变换矩阵,完成六维位姿估计。位置估计可由平移向量直接获得,姿态估计由旋转矩阵间接得到。设最优变换矩阵为:

      则目标从物体坐标系到雷达坐标系的转换关系为:

      式中:M2为ICP配准得到的位姿变换矩阵;M0为雷达坐标系到彩色相机坐标系的外参矩阵;M1为PnP求得的位姿变换矩阵。令:

      其中,旋转矩阵为:${{{{{R}}}}_4}{\text{ = }}\left[ {\begin{array}{*{20}{c}} {{r_{11}}}&{{r_{12}}}&{{r_{13}}} \\ {{r_{21}}}&{{r_{22}}}&{{r_{23}}} \\ {{r_{31}}}&{{r_{32}}}&{{r_{33}}} \end{array}} \right]$ 平移向量${T}_4=\left[\begin{array}{lll}t_1 & t_2 & t_3\end{array}\right]^{\rm{T}}$。传感器6自由度位姿估计为Ψ=(x,y,z,α,β,γ),则位置与平移向量的关系为:

      姿态与旋转矩阵的关系:

    • 在仿真实验中,所需仿真数据为仿真点云数据和仿真图像数据。实物模型如图10(a)所示,尺寸约为500 mm×170 mm×170 mm。根据目标真实尺寸采用相关软件进行三维点云模型创建,并通过在点云模型中增加轴线特征点用于后续运算。点云实物模型和点云模型如图10(b)所示,其中蓝色点为点云模型,红色点为轴线特征点。

      Figure 10.  Physical object and point cloud model

      仿真点云数据通过本实验室提出的三维点云模拟系统,模拟面阵雷达扫描模式采用双切片法[18]提取。假设模型初始位姿如图11所示,所采用的面阵激光雷达和彩色相机为实验室已有设备,参数信息如表1所示。利用1.1节中所提算法对雷达外参信息和彩色相机内参信息进行标定,仿真图像数据通过公式(4)将点云数据转换到图像像素坐标得到。

      Figure 11.  Simulation data

      NameValue
      ModelRS-LD1605 M
      Horizontal field of view/(°)56
      Vertical field of view/(°)31
      Color image resolution2448×1378
      Depth image resolution640×360
      Horizontal angle resolution/(°)0.088
      Vertical angle resolution/(°)0.088
      Detection distance/m50
      Detection accuracy/cm5
      Frame rate/Hz12

      Table 1.  Parameter range of RS-LD1605 M radar

      目标运动模型:飞行轨迹满足参考文献[19]。以目标几何中心建立物体坐标系,绕Y轴旋转角度β为0.64(°)/帧,绕Z轴旋转角度γ为0.51(°)/帧,飞行目标做X轴初速度为20 m/s,Y轴方向自由落体的平抛运动。

      位姿估计指标要求:位置误差≤ 0.01 m;旋转角度误差≤ 5°;解算时间≤ 2 s/幅。

    • 采用实验室自研软件建立仿真环境,按照雷达成像机理模拟激光雷达扫描成像,目标与激光雷达初始物距为30 m。根据可见点判别方法完成对飞行目标进行多视角可测部位点云数据获取。其中部分仿真结果(第20帧、第30帧)如图11所示。由面阵激光雷达成像原理[20]可知,在仿真实验的运动模型下飞行目标在运动过程中可看作静止状态。实验结果表示扫描成像数据输出率可达1 s/帧。(文中算法结果CPU AMD Ryzen 5,在内存为16 GB的PC机上运行所得,操作系统为 Windows 10)。

    • 为检测算法的稳定性,进行50次独立测试对位姿估计结果进行评估。其中部分结果(第20帧、第30帧)如图12图13所示。

      Figure 12.  Display of pose calculation process in frame 20

      Figure 13.  Display of pose calculation process in frame 30

      解算得到的目标点云数据的轴线位置由轴线特征点可知,目标点云数据由3.1节中所提目标运动模型计算可得,然后再由轴线特征点可知目标的轴线位置。通过观察仿真目标点云数据与解算得到的目标点云数据的轴线与YZ轴的夹角来判断旋转角度误差。

      目标位姿数据包括位置参数(Px,Py,Pz)和姿态参数(α,β,γ),其中PxPyPzXYZ轴的位置参数,α为绕X轴旋转角度,β为绕Y轴旋转角度,γ为绕Z轴旋转角度。由于目标为类圆柱形,因此α不作为此次实验结果的参考值。定义估计误差为(ΔPx, ΔPy, ΔPz, Δβ, Δγ)。对三轴平移距离误差及旋转角度误差、算法过程耗时进行了统计分析,平均误差如图14所示。为证明文中算法的高精度性,采用ICP算法对仿真数据进行直接配准,误差结果如图15所示。与参考文献[9-10]所提算法进行对比,比较效果见表2

      Figure 14.  Pose solution error

      Figure 15.  ICP algorithm error

      GroupΔPx/mmΔPy/mmΔPz/mmΔβ/(°)Δγ/(°)Time/ms
      Proposed algorithm1.064.592.070.631.01132
      ICP algorithm2.938.374.210.941.62261
      PnP algorithm1.655.454.110.761.3337
      Algorithm in Ref.[10]2.516.023.880.821.41327

      Table 2.  Comparison of algorithm accuracy

      表2可知,三轴平移误差低于ICP算法和其他算法;文中算法采用PnP算法完成点云粗配准,获取点云间初始旋转平移矩阵;利用IK-D Tree加速临近点搜索,采用ICP算法完成点云精配准。因此运算速度优于ICP算法劣于PnP算法。

      实验结果表明,在不同运动姿态下文中算法均可完成目标位姿估计。所提方法具有较高的估计精度,同时也证明了所提方法的有效性和准确性。

    • 建立半实物仿真试验环境对算法有效性进行验证。由于实验室不具备对炮弹实物进行抛洒的场地条件和抛洒装置条件。遂采用约为3∶1缩放模型的铁质圆柱筒完成半物理实验。目标模型长宽高为160 mm×50 mm×50 mm,激光雷达和彩色相机参数信息与数字仿真试验一致。采用人工抛洒目标的方法模拟目标飞行过程进行了两组试验,目标距离雷达相对距离为[10 m, 11 m]范围内变化,测试不同环境(A组背景为静态建筑物和树木;B组背景为花坛、草地和静态建筑物)、不同光照条件(A组拍摄时间为上午10点,天气状况:晴;B组拍摄时间为下午4点,天气状况:多云)下的文中算法的估计效果。每组试验条件下拍摄图像数量在20张以上,部分拍摄图像如图16所示。

      Figure 16.  Two scene color pictures

    • 为检测图像目标检测算法的稳定性与有效性,对两组图片进行目标提取测试,并与背景差分法、传统ViBe算法以及参考文献[5]所提算法进行比对,对结果进行统计与分析。目标提取结果如图17所示,为定量比较改进算法与原算法的性能,采用了正确率(Precision, P)、召回率(Recall, R)、错误分类比(Percentage of Wrong Classifications, PWC)作为评价指标,定义如下:

      Figure 17.  Comparison of image target extraction algorithms

      式中:TP为正确检测为前景像素的个数;FP为错误检测为前景像素的个数;TN为正确检测为背景像素的个数;FN 为错误检测为背景像素的个数;P为正确判定为前景像素的个数占判定为前景像素总数的百分比;R为正确判定为前景像素的个数占真正前景像素总数的百分比。提取点数以及性能评估见表3~5

      表3~4可见背景差分法在场景存在动态干扰物时提取效果不佳;在背景为静态建筑物居多时,参考文献[5]均有较好的前景目标提取效果,但当背景为动态植物时,背景差分法和参考文献[5]提取效果不佳。ViBe算法在静态背景和动态背景下均有较好的提取效果。但当光线不足且有变化时会产生部分噪点,文中算法召回率相比其他算法略有降低,但在正确率以及错误分类比上都有较好的提升,与传统ViBe算法相比正确率提高了27%,错误分类比降低了36%。背景差分法和参考文献[5]算法直接将运动图像和背景相比,运算量低鲁棒性差且两种方法均为手动设置阈值,因此与文中所提算法相比耗时短;文中算法相比与传统ViBe算法相比,将深度信息加入背景模型样本中作为运动目标检测的权值且采用自适应阈值减少了人工干预,因此耗时有所提升。

      SceneGroupTPFPTNFNPRPWC
      Scene AOur method1 52040336 968 42800.9620.8440.009 4%
      Background subtraction1 6276 015336 372 91730.2130.9040.183%
      Vibe algorithm1 642178336 956 61580.9020.9120.009 9%
      Algorithm in Ref. [5]1 732117336 962 7680.9360.9620.005 5%
      Scene BOur method1 60731336 931 33930.9780.8030.013%
      Background subtraction1 6277 302336 204 23730.1820.8130.228%
      Vibe algorithm1 766725336 861 92340.7090.8830.028%
      Algorithm in Ref. [5]1 9391 007336 833 7610.6580.9690.032%

      Table 3.  Comparison of target extraction accuracy of image

      GroupA/msB/ms
      Proposed algorithm301289
      Background subtraction153147
      Vibe algorithm271253
      Algorithm in Ref.[5]206198

      Table 4.  Time consuming for image target extraction

      Scene numberΔPx/mmΔPy/mmΔPz/mmΔβ/(°)Δγ/(°)
      Scene A2.7015−0.414910.277843.681366.7326
      Scene B1.33680.172310.503682.83294.6933

      Table 5.  Calculation results of target pose

    • 采用第2章中所提算法对目标点云进行提取和位姿估计,结果如图18表6所示。

      单次激光雷达扫描无法直接提取飞行目标的完整轮廓,从而无法掌握目标准确的位姿信息。因此通过彩色图像分别取目标的4个角点和目标每条边的中点作为特征点。利用PnP算法对飞行目标位姿进行两次估计,然后取两次估计结果的平均值与文中位姿求取结果进行对比,来分析实验的有效性。两次PnP算法的位姿求取结果如表7表8所示。求取平均值后与表6所求数据进行对比,误差结果如表9所示。

      Figure 18.  Calculation of target pose in different scenes

      Scene numberΔPx/mmΔPy/mmΔPz/mmΔβ/(°)Δγ/(°)
      Scene A2.7073−0.421310.281844.351865.3863
      Scene B1.33280.168510.507281.75675.3052
      Scene A2.7062−0.413710.274744.272165.8147
      Scene B1.33170.170510.509381.26025.3872

      Table 6.  PnP pose calculation

      Scene numberΔPx/mmΔPy/mmΔPz/mmΔβ/(°)Δγ/(°)
      Scene A5.252.64.50.63−1.13
      Scene B−4.55−2.84.65−1.320.65

      Table 7.  Position and attitude error comparison

      ColourWhiteBlueRed
      Acquisition rate10.970.94

      Table 8.  Point cloud collection rate under different surface colors

      GroupΔPx/mmΔPy/mmΔPz/mmΔβ/(°)Δγ/(°)
      Group A0.974.261.840.580.97
      Group B0.0610.0620.0390.0140.082
      Our algorithm1.064.592.070.631.01

      Table 9.  Number of point clouds at different object distances and surface colors

      半实物仿真试验结果显示,位姿误差小于指标要求。文中算法可以正确有效的对于目标点云进行提取和位置求取,并且能够稳定适用于实际环境之中。

    • 激光雷达对目标进行扫描时,目标物与激光雷达的距离和采集点云数呈负相关;目标表面颜色对脉冲信号的反射强度影响较大。点云反射强度从高到低依次为白色、绿色、蓝色、红色、黑色[21]。目标的表面粗糙度对点云数量和点云反射强度均无明显影响。采用参考文献[21]中提供的目标颜色对点云影响结果,假设白色时目标物的采集率为1,则不同颜色条件下对采集点云数量的影响如表10所示。在文中预设仿真条件下,分别在30 m、15 m和5 m处完成飞行目标点云采集并根据提取率在目标点云中取随机点的方式实现目标不同颜色下的取点模拟,统计不同颜色下采集点的数量,结果如表11所示。

      GroupΔPx/mmΔPy/mmΔPz/mm
      Group A0.974.261.84
      Group B0.0610.0620.039
      Our method1.064.592.07

      Table 10.  Position error

      GroupΔβ/(°)Δγ/(°)
      Group A0.580.97
      Group B0.0140.082
      Our method0.631.01

      Table 11.  Angle error

      表11可知,影响点云采集数量的主要因素为激光雷达距离点云的距离。虽然目标表面颜色也对点云点数有所影响,但由于相差的点云数量过少,在ICP配准时可以忽略不计。

    • 在文中设计算法中,采用改良ViBe算法对目标进行提取时会丢失一部分像素点,经图像形态学处理后可得到完整目标,在恢复完整目标时采用膨胀和腐蚀操作,会使得到的目标与真实目标尺寸略有差别;采用LSD直线检测算法和霍夫变换对图像特征点进行提取时会出现提取误差;采用改进的ICP算法对目标点云进行位姿估计,目标点云的数量影响配准的精度。

    • 针对以上测量误差原因,采用控制变量法设置对照组与文中试验结果进行对比验证。分别设置两个对照组与表2中文中算法造成误差进行比较。A组实验条件为:物距30 m,手动提取图像特征点;B组实验条件为:物距5 m,手动提取图像特征点;

      由文中算法与A组对比可知,采用手动提取图像特征点可以减小误差,但效果不佳;由文中算法与B组对比可知,当激光雷达与目标的距离缩短时,可以有效减少误差。

    • 文中提出一种基于双模态信息融合飞行目标位姿估计算法,选择合适参数的基础上,可有效估计飞行目标的位姿信息。首先使用改进的ViBe算法对图像目标进行提取,较好的克服了传统ViBe算法光照变化不敏感问题,提高了搜索率降低了错误分类比;然后利用点云和图像融合完成对点云目标的快速提取;最后采用PnP算法获取点云间初始旋转平移矩阵,并采用基于IK-D Tree的ICP算法完成目标位姿估计,加快最近点搜索并提高了位姿估计精度。弥补了单源数据难以全面描述运动目标的缺陷,为飞行目标位姿估计提供了一个客观的解决方案。可应用于飞行目标落点精准预测和制导控制,具有较高军事应用价值。

Reference (21)

Catalog

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return