-
过去十年,人工智能得到了快速发展,并应用于各个学科的研究领域当中,基于人工智能色彩迁移的生物医学成像技术是计算机辅助分析与生物成像领域交叉融合的结果,它给传统的生物医学成像领域提供了一个新的发展方向,是一种极具前景的应用技术。文中综述了近年来几种深度学习色彩迁移的技术原理,列举了此类技术在生物医学成像领域中的部分应用,如对组织病理学切片数字图像的色彩迁移、无透镜和单透镜成像的虚拟彩色增强等。按照应用领域、网络结构、学习方法和所解决问题的不同,表1给出了色彩迁移技术的使用情况统计结果。由表1可见,CycleGAN结构和无监督学习类型的深度学习网络在解决各种问题的应用场景中占了较大的比重。如前所述,CycleGAN所表现出的强大性能使其在生物医学成像虚拟染色领域得到广泛应用。其从源域图像到目标域图像强大的转化能力、出色的纹理结构保留能力,使得虚拟染色图像可以有效地应用于组织病理学分析当中。而无监督学习的特点使得研究者无需提供成对的训练样本,减少了研究人员构建符合要求的训练样本数据库的工作,从而大大降低了算法的使用门槛,因此相比监督学习的网络,它拥有着更加广泛的应用。
Application field Network structure Learning method Application problems Supervised learning Unsupervised learning Color transfer technology of pathological section images pix2 pix √ Computational tissue staining Cycle CGAN √ Computational tissue staining CycleGAN √ Mutual stain CycleGAN,
Faster R-CNN√ √ Tissue staining and detection cCGAN,
Residual CycleGAN√ Model improvements for different demand backgrounds Deep-PAM √ Combining different medical image information acquisition technologies GAN √ Unsupervised image style normalization Virtual color enhancement for lensless and single lens imaging GAN √ PIE GAN √ Improvements in lensless microscopes U-Net √ Computational virtual shading method for single lens microscopy Table 1. Statistics on the usage of color transfer technology
尽管这些新的技术方法在处理生物医学成像问题中取得了令人满意的效果,但其在具体应用中尚有问题亟待解决。首先,基于深度学习的色彩迁移网络面临着模型泛化能力较差的问题,网络的泛化能力又极大地依赖于训练数据集的内容丰富性。作为一个“端到端”的复杂非线性映射关系,其结果输出对输入图像的质量要求较高,在不同照明环境下,网络的输出结果很难保持较高的稳定性。而这类应用要求在医学工作中难以完全满足,如切片在不同显微设备间切换观察、切片制作时的薄厚差异等带来的照明差异。这也限制了算法模型被安全有效地应用于临床工作中。另一方面,对于医学图像的色彩迁移结果评价仍未建立合理有效的评判标准,这在一定程度上限制着这类新技术在具体应用中的发展推进。因此,构建更加完备的训练样本库、结合新的信息处理技术、建立起统一的结果评价标准等,将是未来打破限制、发展延伸此类技术的重点。
HTML
[1] | Chantziantoniou N, Donnelly A D, Mukherjee M, et al. Inception and development of the papanicolaou stain method [J]. Acta Cytologica, 2017, 61(4-5): 266-280. |
[2] | Fischer A H, Jacobson K A, Rose J, et al. Hematoxylin and eosin staining of tissue and cell sections [J]. Cold Spring Harbor Protocols, 2008, 2008(5): pdb.prot4986. |
[3] | Beutner E H. Immunofluorescent staining: The fluorescent antibody method [J]. Bacteriological Reviews, 1961, 25(1): 49-76. |
[4] | Irshad H, Veillard A, Roux L, et al. Methods for nuclei detection, segmentation, and classification in digital histopathology: A review—current status and future potential [J]. IEEE Reviews in Biomedical Engineering, 2013, 7: 97-114. |
[5] | Chari S T, Echelmeyer S. Can histopathology be the “Gold Standard” for diagnosing autoimmune pancreatitis? [J]. Gastro-enterology, 2005, 129(6): 2118-2120. |
[6] | Onder D, Zengin S, Sarioglu S. A review on color normalization and color deconvolution methods in histopathology [J]. Applied Immunohistochemistry & Molecular Morphology, 2014, 22(10): 713-719. |
[7] | De Matos J, Britto Jr A S, Oliveira L E S, et al. Histopathologic image processing: A review [J]. arXiv preprint, 2019: 1904.07900. |
[8] | Ursache R, Andersen T G, Marhavý P, et al. A protocol for combining fluorescent proteins with histological stains for diverse cell wall components [J]. The Plant Journal, 2018, 93(2): 399-412. |
[9] | Taqi S A, Sami S A, Sami L B, et al. A review of artifacts in histopathology [J]. Journal of Oral and Maxillofacial Pathology: JOMFP, 2018, 22(2): 279. |
[10] | Celis R, Romero E. Unsupervised color normalisation for H and E stained histopathology image analysis[C]//11th International Symposium on Medical Information Processing and Analysis. International Society for Optics and Photonics, 2015, 9681: 968104. |
[11] | Abraham T, Shaw A, O'Connor D, et al. Slide-free MUSE microscopy to H&E histology modality conversion via unpaired image-to-image translation GAN models [J]. arXiv preprint, 2020: 2008.08579. |
[12] | de Haan K, Rivenson Y, Wu Y, et al. Deep-learning-based image reconstruction and enhancement in optical microscopy [J]. Proceedings of the IEEE, 2019, 108(1): 30-50. |
[13] | Liang H, Plataniotis K N, Li X. Stain style transfer of histopathology images via structure-preserved generative learning[C]//International Workshop on Machine Learning for Medical Image Reconstruction. Cham: Springer, 2020: 153-162. |
[14] | Mahapatra D, Bozorgtabar B, Thiran J P, et al. Structure preserving stain normalization of histopathology images using self supervised semantic guidance[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2020: 309-319. |
[15] | Shi M, McMillan K L, Wu J, et al. Cisplatin nephrotoxicity as a model of chronic kidney disease [J]. Laboratory Investigation, 2018, 98(8): 1105-1121. |
[16] | Rivenson Y, Göröcs Z, Günaydin H, et al. Deep learning microscopy [J]. Optica, 2017, 4(11): 1437-1443. |
[17] | Bilgin C C, Rittscher J, Filkins R, et al. Digitally adjusting chromogenic dye proportions in brightfield microscopy images [J]. Journal of Microscopy, 2012, 245(3): 319-330. |
[18] | Veta M, Pluim J P W, Van Diest P J, et al. Breast cancer histopathology image analysis: A review [J]. IEEE Transactions on Biomedical Engineering, 2014, 61(5): 1400-1411. |
[19] | Hooja S, Pal N, Malhotra B, et al. Comparison of Ziehl Neelsen & Auramine O staining methods on direct and concentrated smears in clinical specimens [J]. The Indian Journal of Tuberculosis, 2011, 58(2): 72-76. |
[20] | Mak K K, Pichika M R. Artificial intelligence in drug development: present status and future prospects [J]. Drug Discovery Today, 2019, 24(3): 773-780. |
[21] | Gunčar G, Kukar M, Notar M, et al. An application of machine learning to haematological diagnosis [J]. Scientific Reports, 2018, 8(1): 1-12. |
[22] | Chen H, Engkvist O, Wang Y, et al. The rise of deep learning in drug discovery [J]. Drug Discovery Today, 2018, 23(6): 1241-1250. |
[23] | Krittanawong C. The rise of artificial intelligence and the uncertain future for physicians [J]. European Journal of Internal Medicine, 2018, 48: e13-e14. |
[24] | Grys B T, Lo D S, Sahin N, et al. Machine learning and computer vision approaches for phenotypic profiling [J]. Journal of Cell Biology, 2017, 216(1): 65-71. |
[25] | Rivenson Y, Liu T, Wei Z, et al. PhaseStain: The digital staining of label-free quantitative phase microscopy images using deep learning [J]. Light: Science & Applications, 2019, 8(1): 1-11. |
[26] | Shaban M T, Baur C, Navab N, et al. Staingan: Stain style transfer for digital histological images[C]//2019 IEEE 16 th International Symposium on Biomedical Imaging (ISBI 2019). IEEE, 2019: 953-956. |
[27] | Lizzi F L, Astor M, Liu T, et al. Ultrasonic spectrum analysis for tissue assays and therapy evaluation [J]. International Journal of Imaging Systems and Technology, 1997, 8(1): 3-10. |
[28] | Linzer M, Norton S J. Ultrasonic tissue characterization [J]. Annual Review of Biophysics and Bioengineering, 1982, 11(1): 303-329. |
[29] | Xu M, Wang L V. Photoacoustic imaging in biomedicine [J]. Review of Scientific Instruments, 2006, 77(4): 041101. |
[30] | Diem M, Chiriboga L, Yee H. Infrared spectroscopy of human cells and tissue. VIII. Strategies for analysis of infrared tissue mapping data and applications to liver tissue [J]. Biopolymers: Original Research on Biomolecules, 2000, 57(5): 282-290. |
[31] | Xiao X, Ma L. Color transfer in correlated color space[C]//Proceedings of the 2006 ACM International Conference on Virtual Reality Continuum and Its Applications, 2006: 305-309. |
[32] | Reinhard E, Adhikhmin M, Gooch B, et al. Color transfer between images [J]. IEEE Computer Graphics and Applications, 2001, 21(5): 34-41. |
[33] | Welsh T, Ashikhmin M, Mueller K. Transferring color to greyscale images[C]//Proceedings of the 29 th Annual Conference on Computer Graphics and Interactive Techniques, 2002: 277-280. |
[34] | Byra M, Galperin M, Ojeda-Fournier H, et al. Breast mass classification in sonography with transfer learning using a deep convolutional neural network and color conversion [J]. Medical Physics, 2019, 46(2): 746-755. |
[35] | Hwang Y, Lee J Y, So Kweon I, et al. Color transfer using probabilistic moving least squares[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014: 3342-3349. |
[36] | Jing Y, Yang Y, Feng Z, et al. Neural style transfer: A review [J]. IEEE Transactions on Visualization and Computer Graphics, 2019, 26(11): 3365-3385. |
[37] | Wan S, Xia Y, Qi L, et al. Automated colorization of a grayscale image with seed points propagation [J]. IEEE Transactions on Multimedia, 2020, 22(7): 1756-1768. |
[38] | Gatys L A, Ecker A S, Bethge M. A neural algorithm of artistic style [J]. arXiv preprint, 2015: 1508.06576. |
[39] | Albawi S, Mohammed T A, Al-Zawi S. Understanding of a convolutional neural network[C]//2017 International Conference on Engineering and Technology (ICET). IEEE, 2017: 1-6. |
[40] | Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 1-9. |
[41] | Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 3431-3440. |
[42] | Huang G, Liu Z, Van Der Maaten L, et al. Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 4700-4708. |
[43] | Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition [J]. arXiv preprint, 2014: 1409.1556. |
[44] | Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural networks [J]. Advances in Neural Information Processing Systems, 2012, 25: 1097-1105. |
[45] | Li Y, Hao Z B, Lei H. Survey of convolutional neural network [J]. Journal of Computer Applications, 2016, 36(9): 2508-2515. |
[46] | Pineda F J. Generalization of back-propagation to recurrent neural networks [J]. Physical Review Letters, 1987, 59(19): 2229. |
[47] | Johnson J, Alahi A, Li Feifei. Perceptual losses for real-time style transfer and super-resolution[C]//European Conference on Computer Vision. Cham: Springer, 2016: 694-711. |
[48] | Liu X C, Cheng M M, Lai Y K, et al. Depth-aware neural style transfer[C]//Proceedings of the Symposium on Non-Photorealistic Animation and Rendering, 2017: 1-10. |
[49] | Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[C]//Advances in Neural Information Processing Systems, 2014 : 2672-2680. |
[50] | Liu H, Fu Z, Han J, et al. Single satellite imagery simultaneous super-resolution and colorization using multi-task deep neural networks [J]. Journal of Visual Communication and Image Representation, 2018, 53: 20-30. |
[51] | Izadyyazdanabadi M, Belykh E, Zhao X, et al. Fluorescence image histology pattern transformation using image style transfer [J]. Frontiers in Oncology, 2019, 9: 519. |
[52] | Rivenson Y, Wang H, Wei Z, et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning [J]. Nature Biomedical Engineering, 2019, 3(6): 466-477. |
[53] | Rivenson Y, de Haan K, Wallace W D, et al. Emerging advances to transform histopathology using virtual staining [J]. BME Frontiers, 2020, 2020: 9647163. |
[54] | Nishar H, Chavanke N, Singhal N. Histopathological stain transfer using style transfer network with adversarial loss[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2020: 330-340. |
[55] | Touvron H, Douze M, Cord M, et al. Powers of layers for image-to-image translation [J]. arXiv preprint, 2020: 2008.05763. |
[56] | Zuo Z, Xu Q, Zhang H, et al. Multimodal image-to-image translation via mutual information estimation and maximization [J]. arXiv preprint, 2020: 2008.03529. |
[57] | Isola P, Zhu J Y, Zhou T, et al. Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 1125-1134. |
[58] | Dai B, Fidler S, Urtasun R, et al. Towards diverse and natural image descriptions via a conditional gan[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 2970-2979. |
[59] | Zhu J Y, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 2223-2232. |
[60] | Almahairi A, Rajeshwar S, Sordoni A, et al. Augmented cyclegan: Learning many-to-many mappings from unpaired data[C]//International Conference on Machine Learning. PMLR, 2018: 195-204. |
[61] | Tschuchnig M E, Oostingh G J, Gadermayr M. Generative adversarial networks in digital pathology: A survey on trends and future potential [J]. Patterns, 2020, 1(6): 100089. |
[62] | Petriceks A H, Olivas J C, Srivastava S. Trends in geriatrics graduate medical education programs and positions, 2001 to 2018 [J]. Gerontology and Geriatric Medicine, 2018, 4: 2333721418777659. |
[63] | Shen D, Wu G, Suk H I. Deep learning in medical image analysis [J]. Annual Review of Biomedical Engineering, 2017, 19: 221-248. |
[64] | Barcia J J. The Giemsa stain: Its history and applications [J]. International Journal of Surgical Pathology, 2007, 15(3): 292-296. |
[65] | Daykin M E, Hussey R S. Staining and histopathological techniques [J]. An Advanced Treatise on Meloidogyne, 1985, 2: 39-48. |
[66] | Pradhan P, Meyer T, Vieth M, et al. Computational tissue staining of non-linear multimodal imaging using supervised and unsupervised deep learning [J]. Biomedical Optics Express, 2021, 12(4): 2280-2298. |
[67] | Barlow H B. Unsupervised learning [J]. Neural Computation, 1989, 1(3): 295-311. |
[68] | Teramoto A, Yamada A, Tsukamoto T, et al. Mutual stain conversion between Giemsa and Papanicolaou in cytological images using cycle generative adversarial network [J]. Heliyon, 2021, 7(2): e06331. |
[69] | Padma S, PV R, Kante R, et al. A comparative study of Staining characteristics of Leishman-Geimsa cocktail and Papanicolaou stain in Cervical Cytology [J]. Asian Pacific Journal of Health Sciences, 2018, 5: 233-236. |
[70] | Gollapudi B, Kamra O P. Applications of a simple Giemsa-staining method in the micronucleus test [J]. Mutat Res, 1979, 64(1): 45-46. |
[71] | Woolman M, Tata A, Bluemke E, et al. An assessment of the utility of tissue smears in rapid cancer profiling with desorption electrospray ionization mass spectrometry (DESI-MS) [J]. Journal of The American Society for Mass Spectrometry, 2016, 28(1): 145-153. |
[72] | Niazi M K K, Parwani A V, Gurcan M N. Digital pathology and artificial intelligence [J]. The Lancet Oncology, 2019, 20(5): e253-e261. |
[73] | Lo Y C, Chung I F, Guo S N, et al. Cycle-consistent GAN-based stain translation of renal pathology images with glomerulus detection application [J]. Applied Soft Computing, 2021, 98: 106822. |
[74] | Khojasteh M, Ward R, MacAulay C. Quantification of membrane IHC stains through multi-spectral imaging[C]//2012 9th IEEE International Symposium on Biomedical Imaging (ISBI). IEEE, 2012: 752-755. |
[75] | Xu Z, Moro C F, Bozóky B, et al. GAN-based virtual re-staining: A promising solution for whole slide image analysis [J]. arXiv preprint, 2019: 1901.04059. |
[76] | de Bel T, Bokhorst J M, van der Laak J, et al. Residual cyclegan for robust domain transformation of histopathological tissue slides [J]. Medical Image Analysis, 2021, 70: 102004. |
[77] | Bornstein M B. Reconstituted rat-tail collagen used as substrate for tissue cultures on coverslips in Maximow slides and roller tubes [J]. Laboratory Investigation, 1958, 7(2): 134-137. |
[78] | Boonstra H, Oosterhuis J W, Oosterhuis A M, et al. Cervical tissue shrinkage by formaldehyde fixation, paraffin wax embedding, section cutting and mounting [J]. Virchows Archiv A, 1983, 402(2): 195-201. |
[79] | Jensen E C. Quantitative analysis of histological staining and fluorescence using ImageJ [J]. The Anatomical Record, 2013, 296(3): 378-381. |
[80] | Kang L, Li X, Zhang Y, et al. Deep learning enables ultraviolet photoacoustic microscopy based histological imaging with near real-time virtual staining [J]. Photoacoustics, 2021, 25: 100308. |
[81] | Baik J W, Kim H, Son M, et al. Intraoperative label-free photoacoustic histopathology of clinical specimens [J]. Laser & Photonics Reviews, 2021, 15(10): 2100124. |
[82] | Morrison D, Harris-Birtill D, Caie P D. Generative deep learning in digital pathology workflows [J]. The American Journal of Pathology, 2021, 191(10): 1717-1723. |
[83] | Zhou N, Cai D, Han X, et al. Enhanced cycle-consistent generative adversarial network for color normalization of H&E stained images[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2019: 694-702. |
[84] | Cho H, Lim S, Choi G, et al. Neural stain-style transfer learning using gan for histopathological images [J]. arXiv preprint, 2017: 1710.08543. |
[85] | Li B, Keikhosravi A, Loeffler A G, et al. Single image super-resolution for whole slide image using convolutional neural networks and self-supervised color normalization [J]. Medical Image Analysis, 2021, 68: 101938. |
[86] | Chen X, Yu J, Cheng S, et al. An unsupervised style normalization method for cytopathology images [J]. Computational and Structural Biotechnology Journal, 2021, 19: 3852-3863. |
[87] | Vahadane A, Peng T, Sethi A, et al. Structure-preserving color normalization and sparse stain separation for histological images [J]. IEEE Transactions on Medical Imaging, 2016, 35(8): 1962-1971. |
[88] | Macenko M, Niethammer M, Marron J S, et al. A method for normalizing histology slides for quantitative analysis[C]//2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. IEEE, 2009: 1107-1110. |
[89] | Khan A M, Rajpoot N, Treanor D, et al. A nonlinear mapping approach to stain normalization in digital histopathology images using image-specific color deconvolution [J]. IEEE Transactions on Biomedical Engineering, 2014, 61(6): 1729-1738. |
[90] | Gupta A, Duggal R, Gehlot S, et al. GCTI-SN: Geometry-inspired chemical and tissue invariant stain normalization of microscopic medical images [J]. Medical Image Analysis, 2020, 65: 101788. |
[91] | Zheng Y, Jiang Z, Zhang H, et al. Adaptive color deconvolution for histological WSI normalization [J]. Computer Methods and Programs in Biomedicine, 2019, 170: 107-120. |
[92] | Shaban M T, Baur C, Navab N, et al. Staingan: Stain style transfer for digital histological images[C]//2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). IEEE, 2019: 953-956. |
[93] | Tellez D, Litjens G, Bándi P, et al. Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology [J]. Medical Image Analysis, 2019, 58: 101544. |
[94] | Bian Y, Jiang Y, Wang J, et al. Deep learning colorful ptychographic iterative engine lens-less diffraction microscopy [J]. Optics and Lasers in Engineering, 2022, 150: 106843. |
[95] | Shen H, Gao J. Deep learning virtual colorful lens-free on-chip microscopy [J]. Chinese Optics Letters, 2020, 18(12): 121705. |
[96] | Bian Y, Jiang Y, Huang Y, et al. Deep learning virtual colorization overcoming chromatic aberrations in singlet lens microscopy [J]. APL Photonics, 2021, 6(3): 031301. |