﻿ 基于置信度的TOF与双目系统深度数据融合<sup>*</sup>
 文章快速检索 高级检索

In-depth data fusion of TOF and stereo vision system based on confidence level
SUN Zhe, ZHANG Yong, CHANG Qutong
School of Instrumentation Science and Opto-electronics Engineering, Beijing University of Aeronautics and Astronautics, Beijing 100083, China
Received: 2017-10-23; Accepted: 2018-01-05; Published online: 2018-02-10 17:11
Corresponding author. ZHANG Yong, E-mail: 06952@buaa.edu.cn
Abstract: To solve the problem of 3D reconstruction in textureless environment like railway, the confidence coefficient is proposed to combine TOF and stereo vision system effectively. Through the joint calibration, the coordinate relationship between TOF and stereo vision systems is established. Then by projecting the points in TOF to left camera in stereo vision system, the disparity map of TOF is obtained. After image segment and surface fitting the disparity map is up-sampled and its resolution is equal to that of stereo images. According to the confidence coefficients of different systems, the system weight values of data fusion are defined. Finally, the proposed method is evaluated with Middlebury dataset, and the results show that the accuracy has been raised twofold or more, and the resolution of disparity map is equal to that of stereo images as well.
Keywords: stereo vision     TOF     data upsampling     data fusion     confidence coefficient

1 双目立体匹配算法 1.1 算法原理

 图 1 双目系统示意图 Fig. 1 Schematic diagram of stereo vision system

P点在左相机坐标系下的坐标为(xL, yL, zL)，在左、右图像上像点对应横坐标为XLXR，则视差d可表示为

 (1)

 (2)

1.2 局部匹配算法

1.2.1 代价计算

 (3)

 (4)

Census转换：

 (5)

1.2.2 代价聚合

 图 2 不同代价聚合算法的视差图对比 Fig. 2 Comparison of disparity maps generated by different cost aggregation algorithms

1.2.3 视差计算

1.2.4 视差优化

1) 左右视差不一致性检测

 (6)

2) 空洞填充

2 TOF系统置信度

TOF相机的测量原理：首先，通过连续发射经过调制的特定频率的光脉冲到被观测物体上；然后，接收从物体反射回去的光脉冲，通过探测光脉冲飞行的往返时间来计算被测物体离相机的距离，同时记录光线强度信息[12]

TOF接收到返回信号的振幅和强度取决于很多因素，其中最重要的是表面反射特性，以及与相机的距离2个方面。虽然也可以直接使用距离定义置信度，但是使用振幅和强度定义置信度考虑了除距离外的其他信息。接收到的振幅对测量的精度影响很大，振幅越大，信噪比越高，因此，测量精度越高。根据文献[1, 13]，TOF相机测得深度值的噪声分布符合高斯标准差：

 (7)

 (8)

 (9)

 (10)
3 融合算法

 图 3 TOF与双目系统数据融合示意图 Fig. 3 Schematic diagram of data fusion of TOF and stereo vision system

1) 数据角度的统一，即从不同相机视角获得的数据通过联合标定，转换到左相机视角下。

2) TOF深度图上采样，生成高分辨率、稠密视差[14]。根据2个系统的数据，逐个像素点进行融合，因此，必须通过一定的插值算法，将低分辨率图像转化为高分辨率图。

3) 融合策略[15]。假设dTdS分别为TOF系统和双目系统的视差，最终生成的视差为d，则有如下关系：

 (11)

3.1 TOF系统高分辨率视差图

TOF相机获取的深度图分辨率较低，因此在TOF深度图和双目匹配得到的视差图融合前，首先对TOF深度图进行稀疏式上采样，步骤如下：

 图 4 TOF系统高分辨率视差图 Fig. 4 Disparity map of TOF system with high resolution

 (12)

3.2 双目系统与TOF系统数据融合

1) 弱纹理区域

2) 非弱纹理区域

 (13)

 (14)

 (15)

4 实验结果与分析

 图 5 样图处理结果示例 Fig. 5 Examples of processing results

TOF系统得到视差图精度高，但分辨率低。首先，将Middlebury数据集的4组数据的视差图同时沿横、纵轴方放大4倍[20]，对其进行高斯滤波；再利用本文方法采样得到高分辨率视差图，与双目立体匹配算法的视差图结果进行融合。以Teddy为例，具体结果如图 6所示。

 图 6 Teddy处理结果示例 Fig. 6 Examples of Teddy processing results

 % 图像 双目立体匹配算法误匹配率 融合算法误匹配率 Teddy 13.34 6.34 Cones 8.56 4.63 Tsukuba 6.52 2.39 Venus 4.79 1.43

5 结论

1) 针对TOF系统，考虑到系统精度与接收到信号振幅的关系，确定了TOF系统的置信度与振幅的关系式；针对双目系统，分为弱纹理区与非弱纹理区，并根据聚合代价的全局最小值与次小值，确定其置信度；即本文提出的融合算法将两系统的优点有效结合。

2) 根据标准数据集的实验结果，不同系统的视差融合之后的精度，比双目立体匹配算法的视差图精度提高了一倍以上，尤其平面区域更加平滑，且分辨率也较TOF系统高。

 [1] ZANUTTIGH P, MARIN G, DAL MUTTO C D, et al. Time-of-flight and structured light depth cameras:Technology and applications[M]. Berlin: Springer, 2016: 23-25. [2] HANSARD M, LEE S, CHOI O, et al. Time-of-flight cameras:Principles, methods and applications[M]. Berlin: Springer, 2013: 50-53. [3] TIPPETTS B, LEE D J, LILLYWHITE K, et al. Review of stereo vision algorithms and their suitability for resource-limited systems[J]. Journal of Real-Time Image Processing, 2016, 11 (1): 5–25. DOI:10.1007/s11554-012-0313-2 [4] YANG Q, AHUJA N, YANG R, et al. Fusion of median and bilateral filtering for range image up-sampling[J]. IEEE Transactions on Image Processing, 2013, 22 (12): 4841–4852. DOI:10.1109/TIP.2013.2278917 [5] GARRO V, MUTTO C D, ZANUTTIGH P, et al. A novel interpolation scheme for range data with side information[C]//Proceedings of the 2009 Conference on Visual Media Production. Piscataway, NJ: IEEE Press, 2009: 52-60. [6] ZHU J, WANG L, GAO J, et al. Spatial-temporal fusion for high accuracy depth maps using dynamic mrfs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32 (5): 899–909. DOI:10.1109/TPAMI.2009.68 [7] VINEET G, JAN C, RADU H. High-resolution depth maps based on ToF-stereo fusion[C]//Proceedings of IEEE Conference on Robotics and Automation. Piscataway, NJ: IEEE Press, 2012: 4742-4749. [8] MUTTO C D, ZANUTTIGH P, MATTOCCIA S, et al. Locally consistent ToF and stereo data fusion[C]//Proceedings of the 12th International Conference on Computer Vision-Volume Part I. Berlin: Springer, 2012: 598-607. [9] FUSIELLO A, TRUCCO E, VERRI A. A compact algorithm for rectification of stereo pairs[J]. Machine Vision and Applications, 2000, 12 (1): 16–22. DOI:10.1007/s001380050120 [10] KAHLMANN T, INGENSAND H. Calibration and development for increased accuracy of 3D range imaging cameras[J]. Journal of Applied Geodesy, 2008, 2 (1): 1–11. DOI:10.1515/JAG.2008.001 [11] SCHARSTEIN D, SZELISKI R. A taxonomy and evaluation of dense twoframe stereo correspondence algorithms[J]. International Journal of Computer Vision, 2002, 47 (1-3): 7–42. [12] PIATTI D, RINAUDO F. SR-4000 and Camcube 3.0 time of flight (ToF) cameras:Tests and comparison[J]. Remote Sensing, 2012, 4 (4): 1069–1089. DOI:10.3390/rs4041069 [13] MUTTO C D, ZANUTTIGH P, CORTELAZZO G. Probabilistic ToF and stereo data fusion based on mixed pixels measurement models[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37 (11): 2260–2272. DOI:10.1109/TPAMI.2015.2408361 [14] DOLSON J, BAEK J, PLAGEMANN C, et al. Up-sampling range data in dynamic environments[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE Press, 2010: 1141-1148. [15] HU X, MORDOHAI P. A quantitative evaluation of confidence measures for stereo vision[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34 (11): 2121–2133. DOI:10.1109/TPAMI.2012.46 [16] GUDMUNDSSON S, AANAES H, LARSEN R. Fusion of stereo vision and time of flight imaging for improved 3D estimation[J]. International Journal Intelligent System Technologies and Applications, 2008, 5 (3): 425–433. [17] EVANGELIDIS G, HANSARD M, HORAUD R. Fusion of range and stereo data for high-resolution scene-modeling[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37 (11): 2178–2192. DOI:10.1109/TPAMI.2015.2400465 [18] NAIR R, LENZEN F, MEISTER S, et al. High accuracy ToF and stereo sensor fusion at interactive rates[C]//Proceedings of European Conference on Computer Vision Workshops. Berlin: Springer, 2012: 7584. [19] 刘娇丽, 李素梅, 李永达, 等. 基于TOF与立体匹配相融合的高分辨率深度获取[J]. 信息技术, 2016 (12): 190–193. LIU J L, LI S M, LI Y D, et al. High-resolution depth maps based on TOF-stereo fusion[J]. Information Technology, 2016 (12): 190–193. (in Chinese) [20] 张康. 基于图像深度感知中的立体匹配和深度增强算法研究[D]. 北京: 清华大学, 2015: 84-87. ZHANG K. Stereo matching and depth enhancement in image-based depth perception[D]. Beijing: Tsinghua University, 2015: 84-87(in Chinese).

#### 文章信息

SUN Zhe, ZHANG Yong, CHANG Qutong

In-depth data fusion of TOF and stereo vision system based on confidence level

Journal of Beijing University of Aeronautics and Astronsutics, 2018, 44(8): 1764-1771
http://dx.doi.org/10.13700/j.bh.1001-5965.2017.0653