﻿ 联合空-谱信息的高光谱影像深度三维卷积网络分类
 文章快速检索 高级检索

Deep 3D convolutional network combined with spatial-spectral features for hyperspectral image classification
LIU Bing , YU Xuchu , ZHANG Pengqiang , TAN Xiong
Information Engineering University, Zhengzhou 450001, China
Foundation support: Key Scientific and Technological Project of Henan Province(No. 152102210014)
First author: LIU Bing(1991-), male, PhD candidate, majors in machine learning and hyperspectral image classification. E-mail:liubing220524@126.com
Abstract: A classification method of hyperspectral images based on deep 3D convolution networks is proposed in order to deal with the high dimensional and small samples of hyperspectral image classification. The method first uses hyperspectral data cube as input, and uses 3D convolution operation to extract 3D spatial-spectral features of hyperspectral data cube. Then, the residual learning is used to construct the deep network and extract higher level feature expression to improve the classification accuracy. Finally, the Dropout regularization method is used to prevent overfitting. Experiments were conducted on the University of Pavia, Indian Pines and Salinas datasets, and the results demonstrate that compared with support vector machine and the existing deep learning classification method for hyperspectral images, the method can effectively improve the classification accuracy of hyperspectral image.
Key words: hyperspectral image classification     convolutional neural network     3D convolution     residual learning

1 本文算法 1.1 卷积神经网络

CNN最初是受到视觉系统中神经机制的启发，针对二维形状的识别而设计的一种多层感知机。该方法将局部连接、权值共享、空间亚采样三种思想结合起来获得某种程度的平移、尺度、形状不变性，具有对二维图像适应性强的特点。同时，CNN结构的可拓展性很强，它通常由若干卷积层、池化层(下采样层)和全连接层组成，可以采用很深的网络结构。因此，CNN能够处理更复杂的分类和识别问题，并取得较为理想的结果。

(1)

1.2 三维卷积

CNN最初是针对二维形状的识别而设计，可以直接处理二维图像，建立从底层信号到高层语义的映射关系，并在视觉图像分类和识别中取得了成功。然而，卷积神经网络在对视频等三维数据进行分析时，具有一定局限性。高光谱遥感影像是三维的数据立方体。因此，卷积神经网络在对高光谱遥感影像地物进行分类前需要使用主成分分析等方法进行降维预处理。但降维处理会损失高光谱图像中的细节信息，而这些细节信息往往有助于区分不同地物类别。

(2)
 图 1 三维卷积操作示意图 Fig. 1 Illustration of 3D convolution

1.3 残差学习

 图 2 残差学习示意 Fig. 2 Illustration of residual learning

1.4 本文网络结构

 图 3 深度三维卷积网络结构 Fig. 3 Architecture of deep 3D convolution network

 层名称 SAE[5] 1D-CNN[6] 2D-CNN[10] 2D-CNN[13] 本文网络 卷积层 0 1 3 1 7 池化层 0 1 3 1 2 全连接层 3 3 1 3 1 总计 3 5 7 5 10

(3)

(4)

2 试验结果与分析

2.1 试验数据

(1) Pavia大学数据集：该数据由ROSIS传感器获得，光谱覆盖范围为430~860 nm，影像大小为610×340像素，空间分辨率为1.3 m，去除受噪声影响的波段后，剩余103个波段可用于分类。该数据集对9种地物进行了标注。地物类别、选取的训练样本数量、确认样本数量以及测试样本数量见表 2

 序号 地物类别 训练样本 确认样本 测试样本 1 柏油路面 180 20 6631 2 草地 180 20 18 649 3 砖块砂砾 180 20 2099 4 树木 180 20 3064 5 金属板 180 20 1345 6 裸土 180 20 5029 7 沥青屋顶 180 20 1330 8 砖块 180 20 3682 9 阴影 180 20 947 总数 1620 180 42 776

(2) Indian Pines数据集：该数据集由AVIRIS传感器获得，光谱覆盖范围为400~2500，影像大小为145×145像素，空间分辨率为20 m，去除受噪声影响的波段后，剩余200个波段可用于分类。该数据集对16种地物进行了标注。参照文献[24]，去除样本数量较少的地物类别，选取样本较多的9种地物进行试验分析。地物类别、选取的训练样本数量、确认样本数量以及测试样本数量见表 3

 序号 地物类别 训练样本 确认样本 测试样本 1 未耕玉米地 180 20 1428 2 玉米幼苗 180 20 830 3 草地/牧场 180 20 483 4 草地树木 180 20 730 5 干草/料堆 180 20 478 6 未耕大豆地 180 20 972 7 大豆幼苗 180 20 2455 8 整理过的大豆 180 20 593 9 木材 180 20 1265 总数 1620 180 9234

(3) Salinas数据集：该数据集由AVIRIS传感器获得，光谱覆盖范围为430~860，影像大小512×217为像素，空间分辨率为3.7 m，去除受噪声影响的波段后，剩余204个波段可用于分类。该数据集对16种地物进行了标注。地物类别、选取的训练样本数量、确认样本数量以及测试样本数量见表 4

 序号 地物类别 训练样本 确认样本 测试样本 1 椰菜_绿_野草1 180 20 2009 2 椰菜_绿_野草2 180 20 3726 3 休耕地 180 20 1976 4 粗糙的休耕地 180 20 1394 5 平滑的休耕地 180 20 2678 6 残株 180 20 3959 7 芹菜 180 20 3579 8 未结果实的葡萄 180 20 11 271 9 正在开发的葡萄园土壤 180 20 6203 10 开始衰老的玉米 180 20 3278 11 长叶莴苣4wk 180 20 1068 12 长叶莴苣5wk 180 20 1927 13 长叶莴苣6wk 180 20 916 14 长叶莴苣7wk 180 20 1070 15 未结果实的葡萄园 180 20 7268 16 葡萄园小路 180 20 1807 总数 2880 320 54 129

2.2 试验结果与分析

 (%) 卷积核数量 Pavia大学数据集 Indian Pines数据集 Salinas数据集 8 93.23 89.06 94.84 16 94.34 93.10 95.46 32 96.30 90.50 93.71 64 92.98 84.32 92.50

 s 卷积核数量 Pavia大学数据集 Indian Pines数据集 Salinas数据集 8 649 1136 2087 16 781 1569 3411 32 1026 3404 6208 64 5069 9471 17 153

 图 4 训练过程中损失函数和总体分类精度在确认样本上的变换情况 Fig. 4 The loss function and the overall accuracy on the validating sample during the training procedure

 (%) 序号 SVM EMPs 1D-CNN 2D-CNN S-3D-CNN 3D-CNN Res-3D-CNN 1 91.24 86.17 76.61 88.80 92.08 95.66 95.23 2 85.37 91.57 95.30 90.33 92.02 89.04 96.88 3 90.66 88.61 81.47 91.19 86.57 95.43 94.85 4 98.07 95.07 94.81 99.71 99.64 98.50 98.43 5 100.0 99.03 99.48 100.0 99.78 99.93 99.85 6 87.25 94.35 58.48 92.22 96.48 93.06 93.28 7 96.32 95.79 91.28 95.56 98.95 98.27 97.67 8 89.65 83.24 80.39 95.17 97.77 95.49 95.71 9 100.0 99.89 99.26 99.58 100.0 100.0 100.0 OA 89.16 91.00 86.17 92.11 93.96 92.96 96.30 AA 93.17 92.64 86.34 94.73 95.92 96.15 96.88 Kappa 85.96 88.24 81.56 89.72 92.12 90.83 95.11

 (%) 序号 SVM EMPs 1D-CNN 2D-CNN S-3D-CNN 3D-CNN Res-3D-CNN 1 82.49 69.75 62.89 83.75 81.93 77.52 86.55 2 85.54 87.71 88.31 96.14 93.25 91.45 94.58 3 96.89 97.93 93.79 98.96 96.69 97.10 98.96 4 99.59 98.77 96.85 98.77 97.26 99.18 99.86 5 100.0 99.79 100.0 98.74 100.0 100.0 100.0 6 82.82 91.87 65.84 93.93 91.05 90.95 94.75 7 73.93 82.57 79.96 77.72 85.74 82.77 86.97 8 91.74 90.56 87.86 98.65 96.29 93.93 96.46 9 98.66 95.10 97.94 98.02 99.92 98.74 99.84 OA 86.34 87.23 82.65 90.00 91.23 89.44 93.10 AA 90.18 90.45 85.94 93.86 93.57 92.40 95.33 Kappa 84.12 85.11 79.69 88.39 89.76 87.68 91.94

 (%) 序号 SVM EMPs 1D-CNN 2D-CNN S-3D-CNN 3D-CNN Res-3D-CNN 1 99.40 99.85 97.61 98.11 100.0 100.0 99.70 2 99.54 99.49 99.62 99.54 98.74 99.97 100.0 3 99.90 100.00 99.75 98.03 99.44 97.27 98.73 4 99.71 99.78 100.0 100.0 100.0 99.64 99.78 5 97.80 97.46 72.63 98.21 99.78 99.78 99.22 6 99.60 99.70 99.82 100.0 100.0 99.87 100.0 7 99.44 98.91 99.27 97.85 99.80 99.66 99.89 8 79.24 81.60 92.41 87.82 85.42 78.14 89.51 9 99.69 98.03 99.79 96.70 98.97 99.40 99.81 10 94.02 97.04 90.09 96.28 97.93 97.10 97.04 11 99.91 98.78 98.50 98.03 100.0 99.25 98.03 12 99.64 100.0 99.90 99.90 100.0 99.95 100.0 13 98.14 99.02 99.67 100.0 100.0 99.78 99.78 14 99.07 99.72 94.21 99.72 100.0 99.91 99.91 15 75.15 85.90 37.07 79.43 79.94 90.00 85.13 16 98.95 99.39 99.28 94.80 99.83 99.23 99.94 OA 91.60 93.54 87.63 93.47 93.89 93.66 95.46 AA 96.20 97.17 92.48 96.53 97.49 97.43 97.90 Kappa 90.67 92.82 86.17 92.73 93.20 92.96 94.95

 图 5 各算法在Pavia大学数据集上的分类结果图及其对应的总体分类精度 Fig. 5 Classification maps and overall accuracy with different methods on the University of Pavia dataset

 图 6 各算法在Indian Pines数据集上的分类结果图及其对应的总体分类精度 Fig. 6 Classification maps and overall accuracy with different methods on the Indian Pines dataset

 图 7 各算法在Salinas数据集上的分类结果图及其对应的总体分类精度 Fig. 7 Classification maps and overall accuracy with different methods on the Salinas dataset

2.3 训练样本数量对分类精度的影响

 图 8 Pavia大学数据集：不同训练样本数目对应的总体分类精度 Fig. 8 Overall accuracy with different number of training samples on the University of Pavia dataset

 图 9 IndianPines数据集：不同训练样本数目对应的总体分类精度 Fig. 9 Overall accuracy with different number of training samples on the Indian Pines dataset

 图 10 Salinas数据集：不同训练样本数目对应的总体分类精度 Fig. 10 Overall accuracy with different number of training samples on the Salinas dataset

 (%) Pavia大学数据集 Indian Pines数据集 Salinas数据集 SVM 68.20 66.21 85.53 EMP 77.90 66.67 86.70 1D-CNN 71.41 41.24 81.33 2D-CNN 79.42 73.86 87.47 Res-3D-CNN 79.49 76.01 87.77

3 总结与展望

﻿

 [1] CAMPS-VALLS G, BRUZZONE L. Kernel-based methods for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2005, 43(6): 1351–1362. DOI:10.1109/TGRS.2005.846154 [2] 唐中奇, 付光远, 陈进, 等. 基于多尺度分割的高光谱图像稀疏表示与分类[J]. 光学精密工程, 2015, 23(9): 2708–2714. TANG Zhongqi, FU Guangyuan, CHEN Jin, et al. Multiscale segmentation-based sparse coding for hyperspectral image classification[J]. Optics and Precision Engineering, 2015, 23(9): 2708–2714. [3] DÓPIDO I, LI Jun, MARPU P R, et al. Semisupervised self-learning for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2013, 51(7): 4032–4044. DOI:10.1109/TGRS.2012.2228275 [4] 田彦平, 陶超, 邹峥嵘, 等. 主动学习与图的半监督相结合的高光谱影像分类[J]. 测绘学报, 2015, 44(8): 919–926. TIAN Yanping, TAO Chao, ZOU Zhengrong, et al. Semi-supervised graph-based hyperspectral image classification with active learning[J]. Acta Geodaetica et Cartographica Sinica, 2015, 44(8): 919–926. DOI:10.11947/j.AGCS.2015.20140221 [5] CHEN Yushi, LIN Zhouhan, ZHAO Xing, et al. Deep learning-based classification of hyperspectral data[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2014, 7(6): 2094–2107. DOI:10.1109/JSTARS.2014.2329330 [6] HU Wei, HUANG Yangyu, LI Wei, et al. Deep convolutional neural networks for hyperspectral image classification[J]. Journal of Sensors, 2015, 2015(7): 220–231. [7] CHEN Yushi, ZHAO Xing, JIA Xiuping. Spectral-spatial classification of hyperspectral data based on deep belief network[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2015, 8(6): 2381–2392. DOI:10.1109/JSTARS.2015.2388577 [8] MOU Lichao, GHAMISI P, ZHU Xiaoxiang. Deep recurrent neural networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(7): 3639–3655. DOI:10.1109/TGRS.2016.2636241 [9] YUE Jun, ZHAO Wenzhi, MAO Shanjun, et al. Spectral-spatial classification of hyperspectral images using deep convolutional neural networks[J]. Remote Sensing Letters, 2015, 6(6): 468–477. DOI:10.1080/2150704X.2015.1047045 [10] YUE Jun, MAO Shanjun, LI Mei. A deep learning framework for hyperspectral image classification using spatial pyramid pooling[J]. Remote Sensing Letters, 2016, 7(9): 875–884. DOI:10.1080/2150704X.2016.1193793 [11] GHAMISI P, CHEN Yushi, ZHU Xiaoxiang. A self-improving convolution neural network for the classification of hyperspectral data[J]. IEEE Geoscience and Remote Sensing Letters, 2016, 13(10): 1537–1541. DOI:10.1109/LGRS.2016.2595108 [12] MEI Shaohui, JI Jingyu, HOU Junhui, et al. Learning sensor-specific spatial-spectral features of hyperspectral images via convolutional neural networks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(8): 4520–4533. DOI:10.1109/TGRS.2017.2693346 [13] LIU Bing, YU Xuchu, ZHANG Pengqiang, et al. Supervised deep feature extraction for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(4): 1909–1921. DOI:10.1109/TGRS.2017.2769673 [14] LIU Peng, ZHANG Hui, EOM K B. Active deep learning for classification of hyperspectral images[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2017, 10(2): 712–724. DOI:10.1109/JSTARS.2016.2598859 [15] LIU Bing, YU Xuchu, ZHANG Pengqiang, et al. A semi-supervised convolutional neural network for hyperspectral image classification[J]. Remote Sensing Letters, 2017, 8(9): 839–848. DOI:10.1080/2150704X.2017.1331053 [16] JIAO Licheng, LIANG Miaomiao, CHEN Huan, et al. Deep fully convolutional network-based spatial distribution prediction for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(10): 5585–5599. DOI:10.1109/TGRS.2017.2710079 [17] JI Shuiwang, XU Wei, YANG Ming, et al. 3D convolutional neural networks for human action recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(1): 221–231. DOI:10.1109/TPAMI.2012.59 [18] 尹宝才, 王文通, 王立春. 深度学习研究综述[J]. 北京工业大学学报, 2015, 41(1): 48–59. YIN Baocai, WANG Wentong, WANG Lichun. Review of deep learning[J]. Journal of Beijing University of Technology, 2015, 41(1): 48–59. [19] HE Kaiming, ZHANG Xianyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778. http://www.tandfonline.com/servlet/linkout?suffix=CIT0020&dbid=16&doi=10.1080%2F15481603.2018.1426091&key=10.1109%2FCVPR.2016.90 [20] NAIR V, HINTON G E. Rectified linear units improve restricted Boltzmann machines[C]//Proceedings of the 27th International Conference on International Conference on Machine Learning. Haifa: Omnipress, 2010: 807-814. https://www.researchgate.net/publication/221345737_Rectified_Linear_Units_Improve_Restricted_Boltzmann_Machines [21] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe: Curran Associates Inc, 2012: 1097-1105. https://www.researchgate.net/publication/267960550_ImageNe [22] 黄鸿, 郑新磊. 高光谱影像空-谱协同嵌入的地物分类算法[J]. 测绘学报, 2016, 45(8): 964–972. HUANG Hong, ZHENG Xinlei. Hyperspectral image land cover classification algorithm based on spatial-spectral coordination embedding[J]. Acta Geodaetica et Cartographica Sinica, 2016, 45(8): 964–972. DOI:10.11947/j.AGCS.2016.20150654 [23] 张春森, 郑艺惟, 黄小兵, 等. 高光谱影像光谱-空间多特征加权概率融合分类[J]. 测绘学报, 2015, 44(8): 909–918. ZHANG Chunsen, ZHENG Yiwei, HUANG Xiaobing, et al. Hyperspectral image classification based on the weighted probabilistic fusion of multiple spectral-spatial features[J]. Acta Geodaetica et Cartographica Sinica, 2015, 44(8): 909–918. DOI:10.11947/j.AGCS.2015.20140544 [24] LI Wei, WU Guodong, ZHANG Fan, et al. Hyperspectral image classification using deep pixel-pair features[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(2): 844–853. DOI:10.1109/TGRS.2016.2616355 [25] BENEDIKTSSON J A, PALMASON J A, SVEINSSON J R. Classification of hyperspectral data from urban areas based on extended morphological profiles[J]. IEEE Transactions on Geoscience and Remote Sensing, 2005, 43(3): 480–491. DOI:10.1109/TGRS.2004.842478 [26] KINGMA D P, BA J. ADAM: A method for Stochastic optimization[R]. The 3rd International Conference for Learning Representations. San Diego: [s.n.], 2015. [27] SRIVASTAVA N, HINTON G, KRIZHEVSKY A, et al. Dropout:a simple way to prevent neural networks from overfitting[J]. The Journal of Machine Learning Research, 2014, 15(1): 1929–1958.
http://dx.doi.org/10.11947/j.AGCS.2019.20170578

0

#### 文章信息

LIU Bing, YU Xuchu, ZHANG Pengqiang, TAN Xiong

Deep 3D convolutional network combined with spatial-spectral features for hyperspectral image classification

Acta Geodaetica et Cartographica Sinica, 2019, 48(1): 53-63
http://dx.doi.org/10.11947/j.AGCS.2019.20170578