Classification and Evaluation of Muti Modal Medical Image Registration Methods and Similarity Measures

Document Type : Review Article

Authors

Faculty of Engineering, Alzahra University, Tehran, Iran

Abstract

Image registration is a fundamental issue in medical image analysis. It refers to the matching process between two or more images using the optimization of a similarity metric to find an optimal transformation function. In recent decades, many studies have been done on the medical image registration topic. Therefore, this paper has investigated four main methodologies to solve the registration problem in medical applications. One of the most important topics in this area is the registration of multi-modal images. In this paper, we have reviewed various multimodal image registration techniques based on deep learning and proposed a classification for these methods. Also one of the essential components of the medical image registration framework is the similarity measure function. There are different similarity metrics in this area and choosing an appropriate measure according to the application is a challenging problem. This paper is to present a review of different similarity measures in medical applications and a classification of these methods. Based on this classification, techniques are investigated and each subclass is evaluated using performance criteria. Therefore, the main goals of this article are as follows: 1) Investigating the most significant image registration approaches. 2) Systematic review of deep learning-based multimodal medical image registration and classify them. 3) Providing classification for various similarity measure techniques according to registration applications. 4) Creating an appropriate platform for evaluating these approaches and introducing the main challenges.

Keywords

References

[1] F.P. Oliveira, J.M.R. Tavares, Medical image registration: a review, Computer methods in biomechanics and biomedical engineering, 17(2) (2014) 73-93.
[2] J. Ma, X. Jiang, A. Fan, J. Jiang, J. Yan, Image matching from handcrafted to deep features: A survey, International Journal of Computer Vision, 129(1) (2021) 23-79.
[3] H. Hermessi, O. Mourali, E. Zagrouba, Multimodal medical image fusion review: Theoretical background and recent advances, Signal Processing, 183 (2021) 108036.
[4] G. Song, J. Han, Y. Zhao, Z. Wang, H. Du, A review on medical image registration as an optimization problem, Current Medical Imaging, 13(3) (2017) 274-283.
[5] Q.R. Razlighi, N. Kehtarnavaz, S. Yousefi, Evaluating similarity measures for brain image registration, Journal of visual communication and image representation, 24(7) (2013) 977-987.
[6] M.A. Viergever, J.A. Maintz, S. Klein, K. Murphy, M. Staring, J.P. Pluim, A survey of medical image registration–under review, in, Elsevier, 2016, pp. 140-144.
[7] M. Wang, P. Li, A review of deformation models in medical image registration, Journal of Medical and Biological Engineering, 39(1) (2019) 1-17.
[8] X. Jiang, J. Ma, G. Xiao, Z. Shao, X. Guo, A review of multimodal image matching: Methods and applications, Information Fusion, 73 (2021) 22-71.
[9] F.E.-Z.A. El-Gamal, M. Elmogy, A. Atwan, Current trends in medical image registration and fusion, Egyptian Informatics Journal, 17(1) (2016) 99-124.
[10] X. Chen, A. Diaz-Pinto, N. Ravikumar, A.F. Frangi, Deep learning in medical image registration, Progress in Biomedical Engineering, 3(1) (2021) 012003.
[11] X. Cao, J. Fan, P. Dong, S. Ahmad, P.-T. Yap, D. Shen, Image registration using machine and deep learning, in:  Handbook of medical image computing and computer assisted intervention, Elsevier, 2020, pp. 319-342.
[12] F. Alam, S.U. Rahman, S. Ullah, K. Gulati, Medical image registration in image guided surgery: Issues, challenges and research opportunities, Biocybernetics and Biomedical Engineering, 38(1) (2018) 71-89.
[13] D. Rivas-Villar, Á.S. Hervella, J. Rouco, J. Novo, Color fundus image registration using a learning-based domain-specific landmark detection methodology, Computers in Biology and Medicine, 140 (2022) 105101.
[14] J. Öfverstedt, J. Lindblad, N. Sladoje, Fast and robust symmetric image registration based on distances combining intensity and spatial information, IEEE Transactions on Image Processing, 28(7) (2019) 3584-3597.
[15] Z. Mohammadi, M.R. Keyvanpour, Similarity Measures in Medical Image Registration A Review Article, in:  2021 12th International Conference on Information and Knowledge Technology (IKT), IEEE, 2021, pp. 89-95.
[16] G. Haskins, U. Kruger, P. Yan, Deep learning in medical image registration: a survey, Machine Vision and Applications, 31(1) (2020) 1-18.
[17] M.-R. Keyvanpour, S. Alehojat, Analytical comparison of learning based methods to increase the accuracy and robustness of registration algorithms in medical imaging,  (2012).
[18] S. Suganyadevi, V. Seethalakshmi, K. Balasamy, A review on deep learning in medical image analysis, International Journal of Multimedia Information Retrieval, 11(1) (2022) 19-38.
[19] K. Aghajani, Multi-modal image registration in the presence of spatially varying intensity distortion using structural representation, Multimedia Tools and Applications, 80(25) (2021) 33885-33909.
[20] F. Zhu, X. Zhu, Z. Huang, M. Ding, Q. Li, X. Zhang, Deep learning based data-adaptive descriptor for non-rigid multi-modal medical image registration, Signal Processing, 183 (2021) 108023.
[21] X. Zhu, M. Ding, T. Huang, X. Jin, X. Zhang, PCANet-based structural representation for nonrigid multimodal medical image registration, Sensors, 18(5) (2018) 1477.
[22] E.M. McKenzie, A. Santhanam, D. Ruan, D. O'Connor, M. Cao, K. Sheng, Multimodality image registration in the head‐and‐neck using a deep learning‐derived synthetic CT as a bridge, Medical physics, 47(3) (2020) 1094-1104.
[23] X. Cao, J. Yang, Y. Gao, Y. Guo, G. Wu, D. Shen, Dual-core steered non-rigid registration for multi-modal images via bi-directional image synthesis, Medical image analysis, 41 (2017) 18-31.
[24] X. Cao, J. Yang, Y. Gao, Q. Wang, D. Shen, Region-adaptive deformable registration of CT/MRI pelvic images via learning-based image synthesis, IEEE Transactions on Image Processing, 27(7) (2018) 3500-3512.
[25] X. Liu, D. Jiang, M. Wang, Z. Song, Image synthesis-based multi-modal image registration framework by using deep fully convolutional networks, Medical & Biological Engineering & Computing, 57(5) (2019) 1037-1048.
[26] R. Han, C.K. Jones, J. Lee, P. Wu, P. Vagdargi, A. Uneri, P.A. Helm, M. Luciano, W.S. Anderson, J.H. Siewerdsen, Deformable MR-CT image registration using an unsupervised, dual-channel network for neurosurgical guidance, Medical image analysis, 75 (2022) 102292.
[27] M. Simonovsky, B. Gutiérrez-Becker, D. Mateus, N. Navab, N. Komodakis, A deep metric for multimodal registration, in:  International conference on medical image computing and computer-assisted intervention, Springer, 2016, pp. 10-18.
[28] X. Cheng, L. Zhang, Y. Zheng, Deep similarity learning for multimodal medical images, Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 6(3) (2018) 248-252.
[29] M. Blendowski, M.P. Heinrich, Combining MRF-based deformable registration and deep binary 3D-CNN descriptors for large lung motion estimation in COPD patients, International journal of computer assisted radiology and surgery, 14(1) (2019) 43-52.
[30] G. Haskins, J. Kruecker, U. Kruger, S. Xu, P.A. Pinto, B.J. Wood, P. Yan, Learning deep similarity metric for 3D MR–TRUS image registration, International journal of computer assisted radiology and surgery, 14(3) (2019) 417-425.
[31] S.K. Zhou, H.N. Le, K. Luu, H.V. Nguyen, N. Ayache, Deep reinforcement learning in medical imaging: A literature review, Medical image analysis, 73 (2021) 102193.
[32] J. Hu, Z. Luo, X. Wang, S. Sun, Y. Yin, K. Cao, Q. Song, S. Lyu, X. Wu, End-to-end multimodal image registration via reinforcement learning, Medical Image Analysis, 68 (2021) 101878.
[33] K. Ma, J. Wang, V. Singh, B. Tamersoy, Y.-J. Chang, A. Wimmer, T. Chen, Multimodal image registration with deep context reinforcement learning, in:  International conference on medical image computing and computer-assisted intervention, Springer, 2017, pp. 240-248.
[34] S. Miao, S. Piat, P. Fischer, A. Tuysuzoglu, P. Mewes, T. Mansi, R. Liao, Dilated FCN for multi-agent 2D/3D medical image registration, in:  Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
[35] H. Xiao, X. Teng, C. Liu, T. Li, G. Ren, R. Yang, D. Shen, J. Cai, A review of deep learning-based three-dimensional medical image registration methods, Quantitative Imaging in Medicine and Surgery, 11(12) (2021) 4895.
[36] S. Abbasi, M. Tavakoli, H.R. Boveiri, M.A.M. Shirazi, R. Khayami, H. Khorasani, R. Javidan, A. Mehdizadeh, Medical image registration using unsupervised deep neural network: A scoping literature review, Biomedical Signal Processing and Control, 73 (2022) 103444.
[37] S.S.M. Salehi, S. Khan, D. Erdogmus, A. Gholipour, Real-time deep pose estimation with geodesic loss for image-to-template rigid registration, IEEE transactions on medical imaging, 38(2) (2018) 470-481.
[38] X. Wang, L. Mao, X. Huang, M. Xia, Z. Gu, Multimodal MR image registration using weakly supervised constrained affine network, Journal of Modern Optics, 68(13) (2021) 679-688.
[39] J. Fan, X. Cao, P.-T. Yap, D. Shen, BIRNet: Brain image registration using dual-supervised fully convolutional networks, Medical image analysis, 54 (2019) 193-206.
[40] H. Xiao, R. Ni, S. Zhi, W. Li, C. Liu, G. Ren, X. Teng, W. Liu, W. Wang, Y. Zhang, A dual‐supervised deformation estimation model (DDEM) for constructing ultra‐quality 4D‐MRI based on a commercial low‐quality 4D‐MRI for liver cancer radiation therapy, Medical Physics, 49(5) (2022) 3159-3170.
[41] B.D. De Vos, F.F. Berendsen, M.A. Viergever, H. Sokooti, M. Staring, I. Išgum, A deep learning framework for unsupervised affine and deformable image registration, Medical image analysis, 52 (2019) 128-143.
[42] Y. Ma, D. Niu, J. Zhang, X. Zhao, B. Yang, C. Zhang, Unsupervised deformable image registration network for 3D medical images, Applied Intelligence, 52(1) (2022) 766-779.
[43] S. Li, T. Zhang, D. Zhang, Y. Nie, J. Wang, Metric learning for patch-based 3-d image registration, IEEE Transactions on Automation Science and Engineering, 16(4) (2019) 1575-1583.
[44] M.I. Gandhi, Image registration quality assessment with similarity measures-a research study, in:  2015 International Conference on Communications and Signal Processing (ICCSP), IEEE, 2015, pp. 0084-0088.
[45] Z. Chen, Z. Xu, Q. Gui, X. Yang, Q. Cheng, W. Hou, M. Ding, Self-learning based medical image representation for rigid real-time and multimodal slice-to-volume registration, Information Sciences, 541 (2020) 502-515.
[46] A.A. Goshtasby, Image registration: Principles, tools and methods, Springer Science & Business Media, 2012.
[47] S. Liu, B. Yang, Y. Wang, J. Tian, L. Yin, W. Zheng, 2D/3D multimode medical image registration based on normalized cross-correlation, Applied Sciences, 12(6) (2022) 2828.
[48] M.A. Azam, K.B. Khan, M. Ahmad, M. Mazzara, Multimodal medical image registration and fusion for quality Enhancement, Cmc-Comput. Mater. Contin, 68 (2021) 821-840.
[49] H. Bakhshayesh, S.P. Fitzgibbon, A.S. Janani, T.S. Grummett, K.J. Pope, Detecting synchrony in EEG: A comparative study of functional connectivity measures, Computers in biology and medicine, 105 (2019) 1-15.
[50] D. Sengupta, P. Gupta, A. Biswas, A survey on mutual information based medical image registration algorithms, Neurocomputing, 486 (2022) 174-188.
[51] L. Gong, H. Wang, C. Peng, Y. Dai, M. Ding, Y. Sun, X. Yang, J. Zheng, Non-rigid MR-TRUS image registration for image-guided prostate biopsy using correlation ratio-based mutual information, Biomedical engineering online, 16(1) (2017) 1-21.
[52] H. Rivaz, Z. Karimaghaloo, D.L. Collins, Self-similarity weighted mutual information: a new nonrigid image registration metric, Medical image analysis, 18(2) (2014) 343-358.
[53] Y. Fu, Y. Lei, T. Wang, W.J. Curran, T. Liu, X. Yang, Deep learning in medical image registration: a review, Physics in Medicine & Biology, 65(20) (2020) 20TR01.
[54] R.W. So, A.C. Chung, Multi-modal non-rigid image registration based on similarity and dissimilarity with the prior joint intensity distributions, in:  2010 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, IEEE, 2010, pp. 368-371.
[55] S. Jabari, M. Rezaee, F. Fathollahi, Y. Zhang, Multispectral change detection using multivariate Kullback-Leibler distance, ISPRS Journal of Photogrammetry and Remote sensing, 147 (2019) 163-177.
[56] R.W. So, A.C. Chung, A novel learning-based dissimilarity metric for rigid and non-rigid medical image registration by using Bhattacharyya Distances, Pattern Recognition, 62 (2017) 161-174.
[57] X. Chen, X. Wang, K. Zhang, K.-M. Fung, T.C. Thai, K. Moore, R.S. Mannel, H. Liu, B. Zheng, Y. Qiu, Recent advances and clinical applications of deep learning in medical image analysis, Medical Image Analysis,  (2022) 102444.
[58] N. Zhu, M. Najafi, B. Han, S. Hancock, D. Hristov, Feasibility of image registration for ultrasound-guided prostate radiotherapy based on similarity measurement by a convolutional neural network, Technology in cancer research & treatment, 18 (2019) 1533033818821964.
[59] M. Blendowski, L. Hansen, M.P. Heinrich, Weakly-supervised learning of multi-modal features for regularised iterative descent in 3D image registration, Medical image analysis, 67 (2021) 101822.
[60] B.D.d. Vos, F.F. Berendsen, M.A. Viergever, M. Staring, I. Išgum, End-to-end unsupervised deformable image registration with a convolutional neural network, in:  Deep learning in medical image analysis and multimodal learning for clinical decision support, Springer, 2017, pp. 204-212.
[61] G. Balakrishnan, A. Zhao, M.R. Sabuncu, J. Guttag, A.V. Dalca, VoxelMorph: a learning framework for deformable medical image registration, IEEE transactions on medical imaging, 38(8) (2019) 1788-1800.
[62] Y. Zheng, S. Jiang, Z. Yang, Deformable registration of chest CT images using a 3D convolutional neural network based on unsupervised learning, Journal of Applied Clinical Medical Physics, 22(10) (2021) 22-35.
[63] Y. Lei, Y. Fu, T. Wang, Y. Liu, P. Patel, W.J. Curran, T. Liu, X. Yang, 4D-CT deformable image registration using multiscale unsupervised deep learning, Physics in Medicine & Biology, 65(8) (2020) 085003.
[64] G.A. Benvenuto, M. Colnago, W. Casaca, Unsupervised Deep Learning Network for Deformable Fundus Image Registration, in:  ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2022, pp. 1281-1285.
[65] S.K. Zhou, H. Greenspan, C. Davatzikos, J.S. Duncan, B. Van Ginneken, A. Madabhushi, J.L. Prince, D. Rueckert, R.M. Summers, A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises, Proceedings of the IEEE, 109(5) (2021) 820-838.
[66] H. Siebert, L. Hansen, M.P. Heinrich, Learning a Metric for Multimodal Medical Image Registration without Supervision Based on Cycle Constraints, Sensors, 22(3) (2022) 1107.
AUT Journal of Modeling and Simulation
Volume 55, Issue 1
June 2023
Pages 53-70

Files

Share

How to cite

Statistics