Hyperspectral images reconstruction using adversarial networks from single RGB image

Liu Pengfei; Zhao Huaici; Li Peixuan

doi:10.3788/IRLA20200093

Volume 49 Issue S1

Sep. 2020

Turn off MathJax

Article Contents

Article Navigation > Infrared and Laser Engineering > 2020 > 49(S1): 20200093

Liu Pengfei, Zhao Huaici, Li Peixuan. Hyperspectral images reconstruction using adversarial networks from single RGB image[J]. Infrared and Laser Engineering, 2020, 49(S1): 20200093. doi: 10.3788/IRLA20200093

Citation:

Liu Pengfei, Zhao Huaici, Li Peixuan. Hyperspectral images reconstruction using adversarial networks from single RGB image[J]. Infrared and Laser Engineering, 2020, 49(S1): 20200093. doi: 10.3788/IRLA20200093

Hyperspectral images reconstruction using adversarial networks from single RGB image

doi: 10.3788/IRLA20200093

Liu Pengfei^1,2,3,4,5,
Zhao Huaici^{1,2,4,5
,
,},
Li Peixuan^1,2,3,4,5

1. Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China;
2. Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110169, China;
3. University of Chinese Academy of Sciences, Beijing 100049, China;
4. Key Laboratory of Optical-Electronics Information Processing, Chinese Academy of Sciences, Shenyang 110016, China;
5. The Key Lab of Image Understanding and Computer Vision, Shenyang 110016, China

Received Date: 2020-04-02
Rev Recd Date: 2020-05-05
Publish Date: 2020-09-22

Abstract

Hyperspectral imaging can provide more spectral information than an ordinary RGB camera. The spectral information has been beneficial to numerous applications, such as monitoring natural environment changes and classifying plants and soils in agriculture. The hyperspectral images reconstruction from a single RGB image is severely unconstrained problem. Previous methods need additional components or the spectral response by commercial camera. An end-to-end conditional generative adversarial network was proposed with modified residual network as backbone. The feature pyramid was used inside the network and a scale attention module was designed to fuse local and global information. In order to provide more accurate solution, another distinct architecture was proposed, named WNet. Experiments manifested the superiority of the proposed method over other representative methods in terms of quality and quantity. Experiments used both synthesized RGB images using public hyperspectral data and real-world image by ordinary camera demonstrate that proposed method outperforms the state-of-the-art. The WNet drops 45% and 50% in terms of RMSE and relative RMSE on the ICVL dataset than sparse coding.
- hyperspectral imaging,
- feature pyramid,
- generative adversarial network,
- attention mechanism

References

[1]	Huang Hong, Tang Yuxiao, Duan Yule. Feature extraction of hyperspectral image with semi-supervised multi-graph embedding[J]. Optics and Precision Engineering, 2020, 28(2):443-456. (in Chinese)黄鸿, 唐玉枭, 段宇乐. 半监督多图嵌入的高光谱影像特征提取[J]. 光学精密工程, 2020, 28(2):443-456.
[2]	Zhao Huijie, Li Jimin, Jia Guorui, et al. Correlation-wavelet method for separation of hyperspectral thermal infrared temperature and emissivity[J]. Optics and Precision Engineering, 2019, 27(8):1738-1744. (in Chinese)赵慧洁, 李济民, 贾国瑞, 等. 高光谱热红外温度发射率分离的相关小波法[J]. 光学精密工程, 2019, 27(8):1738-1744.
[3]	He Sailing, Chen Xiang, Li Shuo, et al. Small hyperspectral imagers and lidars and their marine applications[J]. Infrared and Laser Engineering, 2020, 49(2):0203001. (in Chinese)何赛灵, 陈祥, 李硕, 等. 小型高光谱图谱仪与激光雷达及其海洋应用[J].红外与激光工程, 2020, 49(2):0203001.
[4]	Arad B, Benshahar O. Sparse recovery of hyperspectral signal from natural RGB images[C]//European Conference on Computer Vision, 2016:19-34.
[5]	Zhang Zexia, Chang Jun, Ren Hongxi, et al. Snapshot imaging spectrometer based on a microlens array[J]. Chinese Optics Letters, 2019, 17(1):35-39.
[6]	Takatani T, Aoto T, Mukaigawa Y, et al. One-shot hyperspectral imaging using faced reflectors[C]//Computer Vision and Pattern Recognition, 2017:2692-2700.
[7]	Oh S W, Brown M S, Pollefeys M, et al. Do it yourself hyperspectral imaging with everyday digital cameras[C]//Computer Vision and Pattern Recognition, 2016:2461-2469.
[8]	Akhtar N, Shafait F, Mian A, et al. Hierarchical beta Process with gaussian process prior for hyperspectral image super resolution[C]//European Conference on Computer Vision, 2016:103-120.
[9]	Jia Y, Zheng Y, Gu L, et al. From RGB to spectrum for natural scenes via manifold-based mapping[C]//International Conference on Computer Vision, 2017:4715-4723.
[10]	Wang Chunzhe, An Junshe, Jiang Xiujie, et al. Region proposal optimization algorithm based on convolutional neural networks[J]. Chinese Optics, 2019, 12(6):1348-1361. (in Chinese)王春哲, 安军社, 姜秀杰, 等. 基于卷积神经网络的候选区域优化算法[J]. 中国光学, 2019, 12(6):1348-1361.
[11]	Zhang Lamei, Chen Zexi, Zou Bin. Fine classification of polarimetric SAR images based on 3D convolutional neural network[J]. Infrared and Laser Engineering, 2018, 47(7):0703001. (in Chinese)张腊梅, 陈泽茜, 邹斌. 基于3D卷积神经网络的PolSAR图像精细分类[J]. 红外与激光工程, 2018, 47(7):0703001.
[12]	Zhang Xiu, Zhou Wei, Duan Zhemin, et al. Convolutional sparse auto-encoder for image super-resolution reconstruction[J]. Infrared and Laser Engineering, 2019, 48(1):0126005. (in Chinese)张秀, 周巍, 段哲民, 等. 基于卷积稀疏自编码的图像超分辨率重建[J].红外与激光工程, 2019, 48(1):0126005.
[13]	Xiong Z, Shi Z, Li H, et al. HSCNN:CNN-based hyperspectral image recovery fromspectrally undersampled projections[C]//International Conference on Computer Vision, 2017:518-525.
[14]	Shi Z, Chen C, Xiong Z, et al. HSCNN+:advanced CNN-based hyperspectral recovery from RGB images[C]//Computer Vision and Pattern Recognition, 2018:939-947.
[15]	Alvarezgila A, De Weijer J V, Garrote E, et al. Adversarial networks for spatial context-aware spectral image reconstruction from RGB[C]//International Conference on Computer Vision, 2017:480-490.
[16]	Chakrabarti A, Zickler T. Statistics of real-world hyperspectral images[C]//Computer Vision and Pattern Recognition, 2011:193-200.

Proportional views

通讯作者: 陈斌, bchen63@163.com

1.
沈阳化工大学材料科学与工程学院沈阳 110142

Get Citation

PDF

XML

Article Metrics

Article views(526) PDF downloads(56) Cited by()

Proportional views

Hyperspectral images reconstruction using adversarial networks from single RGB image

1. Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China;

2. Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110169, China;

3. University of Chinese Academy of Sciences, Beijing 100049, China;

4. Key Laboratory of Optical-Electronics Information Processing, Chinese Academy of Sciences, Shenyang 110016, China;

5. The Key Lab of Image Understanding and Computer Vision, Shenyang 110016, China

Received Date: 2020-04-02

Rev Recd Date: 2020-05-05

Publish Date: 2020-09-22

Key words:

Abstract: Hyperspectral imaging can provide more spectral information than an ordinary RGB camera. The spectral information has been beneficial to numerous applications, such as monitoring natural environment changes and classifying plants and soils in agriculture. The hyperspectral images reconstruction from a single RGB image is severely unconstrained problem. Previous methods need additional components or the spectral response by commercial camera. An end-to-end conditional generative adversarial network was proposed with modified residual network as backbone. The feature pyramid was used inside the network and a scale attention module was designed to fuse local and global information. In order to provide more accurate solution, another distinct architecture was proposed, named WNet. Experiments manifested the superiority of the proposed method over other representative methods in terms of quality and quantity. Experiments used both synthesized RGB images using public hyperspectral data and real-world image by ordinary camera demonstrate that proposed method outperforms the state-of-the-art. The WNet drops 45% and 50% in terms of RMSE and relative RMSE on the ICVL dataset than sparse coding.

HTML

Reference (16)

[1]	Huang Hong, Tang Yuxiao, Duan Yule. Feature extraction of hyperspectral image with semi-supervised multi-graph embedding[J]. Optics and Precision Engineering, 2020, 28(2):443-456. (in Chinese)黄鸿, 唐玉枭, 段宇乐. 半监督多图嵌入的高光谱影像特征提取[J]. 光学精密工程, 2020, 28(2):443-456.
[2]	Zhao Huijie, Li Jimin, Jia Guorui, et al. Correlation-wavelet method for separation of hyperspectral thermal infrared temperature and emissivity[J]. Optics and Precision Engineering, 2019, 27(8):1738-1744. (in Chinese)赵慧洁, 李济民, 贾国瑞, 等. 高光谱热红外温度发射率分离的相关小波法[J]. 光学精密工程, 2019, 27(8):1738-1744.
[3]	He Sailing, Chen Xiang, Li Shuo, et al. Small hyperspectral imagers and lidars and their marine applications[J]. Infrared and Laser Engineering, 2020, 49(2):0203001. (in Chinese)何赛灵, 陈祥, 李硕, 等. 小型高光谱图谱仪与激光雷达及其海洋应用[J].红外与激光工程, 2020, 49(2):0203001.
[4]	Arad B, Benshahar O. Sparse recovery of hyperspectral signal from natural RGB images[C]//European Conference on Computer Vision, 2016:19-34.
[5]	Zhang Zexia, Chang Jun, Ren Hongxi, et al. Snapshot imaging spectrometer based on a microlens array[J]. Chinese Optics Letters, 2019, 17(1):35-39.
[6]	Takatani T, Aoto T, Mukaigawa Y, et al. One-shot hyperspectral imaging using faced reflectors[C]//Computer Vision and Pattern Recognition, 2017:2692-2700.
[7]	Oh S W, Brown M S, Pollefeys M, et al. Do it yourself hyperspectral imaging with everyday digital cameras[C]//Computer Vision and Pattern Recognition, 2016:2461-2469.
[8]	Akhtar N, Shafait F, Mian A, et al. Hierarchical beta Process with gaussian process prior for hyperspectral image super resolution[C]//European Conference on Computer Vision, 2016:103-120.
[9]	Jia Y, Zheng Y, Gu L, et al. From RGB to spectrum for natural scenes via manifold-based mapping[C]//International Conference on Computer Vision, 2017:4715-4723.
[10]	Wang Chunzhe, An Junshe, Jiang Xiujie, et al. Region proposal optimization algorithm based on convolutional neural networks[J]. Chinese Optics, 2019, 12(6):1348-1361. (in Chinese)王春哲, 安军社, 姜秀杰, 等. 基于卷积神经网络的候选区域优化算法[J]. 中国光学, 2019, 12(6):1348-1361.
[11]	Zhang Lamei, Chen Zexi, Zou Bin. Fine classification of polarimetric SAR images based on 3D convolutional neural network[J]. Infrared and Laser Engineering, 2018, 47(7):0703001. (in Chinese)张腊梅, 陈泽茜, 邹斌. 基于3D卷积神经网络的PolSAR图像精细分类[J]. 红外与激光工程, 2018, 47(7):0703001.
[12]	Zhang Xiu, Zhou Wei, Duan Zhemin, et al. Convolutional sparse auto-encoder for image super-resolution reconstruction[J]. Infrared and Laser Engineering, 2019, 48(1):0126005. (in Chinese)张秀, 周巍, 段哲民, 等. 基于卷积稀疏自编码的图像超分辨率重建[J].红外与激光工程, 2019, 48(1):0126005.
[13]	Xiong Z, Shi Z, Li H, et al. HSCNN:CNN-based hyperspectral image recovery fromspectrally undersampled projections[C]//International Conference on Computer Vision, 2017:518-525.
[14]	Shi Z, Chen C, Xiong Z, et al. HSCNN+:advanced CNN-based hyperspectral recovery from RGB images[C]//Computer Vision and Pattern Recognition, 2018:939-947.
[15]	Alvarezgila A, De Weijer J V, Garrote E, et al. Adversarial networks for spatial context-aware spectral image reconstruction from RGB[C]//International Conference on Computer Vision, 2017:480-490.
[16]	Chakrabarti A, Zickler T. Statistics of real-world hyperspectral images[C]//Computer Vision and Pattern Recognition, 2011:193-200.

Hyperspectral images reconstruction using adversarial networks from single RGB image

doi: 10.3788/IRLA20200093

Abstract

References

Proportional views

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Related

Proportional views