[en] Super-resolution (SR) techniques present a suitable solution to increase the image resolution acquired using an ultrasound device characterized by a low image resolution. This can be particularly beneficial in low-resource imaging settings. This work surveys advanced SR techniques applied to enhance the resolution and quality of fetal ultrasound images, focusing Dual back-projection based internal learning (DBPISR) technique, which utilizes internal learning for blind super-resolution, as opposed to blind super-resolution generative adversarial network (BSRGAN), real-world enhanced super-resolution generative adversarial network (Real-ESRGAN), swin transformer for image restoration (SwinIR) and SwinIR-Large. The dual back-projection approach enhances SR by iteratively refining downscaling and super-resolution processes through a dual network training method, achieving high accuracy in kernel estimation and image reconstruction. Real-ESRGAN uses synthetic data to simulate complex real-world degradations, incorporating a U-shaped network (U-Net) discriminator to improve training stability and visual performance. BSRGAN addresses the limitations of traditional degradation models by introducing a realistic and comprehensive degradation process involving blur, downsampling, and noise, leading to superior real-world SR performance. Swin models (SwinIR and SwinIR_large) employ a Swin Transformer architecture for image restoration, excelling in capturing long-range dependencies and complex structures, resulting in an outstanding performance in PSNR, SSIM, NIQE, and BRISQUE metrics. The tested images, sourced from five developing countries and often of lower quality, enabled us to show that these approaches can help enhance the quality of the images. Evaluations on fetal ultrasound images reveal that these methods significantly enhance image quality, with DBPISR, Real-ESRGAN, BSRGAN, SwinIR, and SwinIR-Large showing notable improvements in PSNR and SSIM, thereby highlighting their potential for improving the resolution and diagnostic utility of fetal ultrasound images. We evaluated the five aforementioned Super-Resolution models, analyzing their impact on both image quality and classification tasks. Our findings indicate that these techniques hold great potential for enhancing the evaluation of medical images, particularly in development countries. Among the models tested, Real-ESRGAN consistently enhanced both image quality and diagnostic accuracy, even when challenged by limited and variable datasets. This finding was further supported by deploying the ConvNext-base classifier, which demonstrated improved performance when applied to the super-resolved images. Real-ESRGAN's capacity to enhance image quality, and in turn, classification accuracy, highlights its potential to address the resource constraints often encountered in these settings.
Disciplines :
Computer science
Author, co-author :
Boumeridja, Hafida; LIST Laboratory, University M'Hamed Bougara, Boumerdes, Algeria
Ammar, Mohammed; LIST Laboratory, University M'Hamed Bougara, Boumerdes, Algeria. m.ammar@univ-boumerdes.dz
Alzubaidi, Mahmood; College of Science and Engineering, Hamad Bin Khalifa University, Doha, 34110, Qatar. Malzubaidi@hbku.edu.qa
Mahmoudi, Saïd ; Université de Mons - UMONS > Faculté Polytechnique > Service Informatique, Logiciel et Intelligence artificielle
Benamer, Lamya Nawal; Obstetrics and Gynecology Department, School of Medicine, Algiers University, Algiers, Algeria
Agus, Marco; College of Science and Engineering, Hamad Bin Khalifa University, Doha, 34110, Qatar
Househ, Mowafa; College of Science and Engineering, Hamad Bin Khalifa University, Doha, 34110, Qatar
Lekadir, Karim; Artificial Intelligence in Medicine Lab (BCN-AIM), Department of Mathematics and Computer Science, Universitat de Barcelona, Barcelona, Spain ; Institució Catalana de Recerca i Estudis Avançats (ICREA), Passeig Lluís Companys 23, Barcelona, Spain
El Habib Daho, Mostafa; Univ Bretagne Occidentale, Brest, 29200, France ; Inserm, UMR 1101, Brest, 29200, France
Language :
English
Title :
Enhancing fetal ultrasound image quality and anatomical plane recognition in low-resource settings using super-resolution models.
F114 - Informatique, Logiciel et Intelligence artificielle
Research institute :
R300 - Institut de Recherche en Technologies de l'Information et Sciences de l'Informatique
Funders :
Hamad Bin Khalifa University, Qatar
Funding text :
This publication was funded by the PPM-7th Cycle grant (PPM 07-0409-240041, AMAL-For-Qatar) from the Qatar National Research Fund, a member of the Qatar Foundation. The project was supported through a subcontract agreement between the American University of Beirut (AUB) and Hamad Bin Khalifa University (HBKU) for the project titled \u201CFADA\u201D (AUB Reference: 104348). This research was partly funded by a grant from the European Research Council (ERC) under the European Union\u2019s Horizon Europe research and innovation programme (AIMIX project - Grant Agreement No. 101044779).
S. Suganyadevi V. Seethalakshmi K. Balasamy A review on deep learning in medical image analysis Int. J. Multimedia Inf. Retrieval 11 19 38 1:STN:280:DC%2BB2crms12jsg%3D%3D 10.1007/s13735-021-00218-1 1518.05011
Y. Zhao X. Wang T. Che G. Bao S. Li Multi-task deep learning for medical image computing and analysis: A review Comput. Biol. Med. 153 10.1016/j.compbiomed.2022.106496 36634599 1399.68030 106496
Alzubaidi, M. et al. Toward deep observation: A systematic survey on artificial intelligence techniques to monitor fetus via ultrasound images. iScience25, 104713, https://doi.org/10.1016/j.isci.2022.104713 (2022).
M.C. Fiorentino F.P. Villani M. Di Cosmo E. Frontoni S. Moccia A review on deep-learning algorithms for fetal ultrasound-image analysis Med. Image Anal. 83 10.1016/j.media.2022.102629 36308861 102629
Leung, K.-Y. Applications of advanced ultrasound technology in obstetrics. Diagnostics11, https://doi.org/10.3390/diagnostics11071217 (2021).
Alzubaidi, M. et al. Ensemble transfer learning for fetal head analysis: From segmentation to gestational age and weight prediction. Diagnostics12, https://doi.org/10.3390/diagnostics12092229 (2022).
Alzubaidi, M. et al. A composite image processing technique to enhance segmentation of ultrasound images. In Proceedings of the 2022 5th International Conference on Digital Medicine and Image Processing, DMIP ’22, 1–7, https://doi.org/10.1145/3576938.3576939 (Association for Computing Machinery, New York, NY, USA, 2023).
Dong, C., Loy, C. C., He, K. & Tang, X. Learning a deep convolutional network for image super-resolution. In Fleet, D., Pajdla, T., Schiele, B. & Tuytelaars, T. (eds.) Computer Vision – ECCV 2014, 184–199 (Springer International Publishing, Cham, 2014).
C. Dong C.C. Loy K. He X. Tang Image super-resolution using deep convolutional networks IEEE Trans. Pattern Anal. Mach. Intell. 38 295 307 10.1109/TPAMI.2015.2439281 26761735 0977.60079
Shocher, A., Cohen, N. & Irani, M. Zero-shot super-resolution using deep internal learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3118–3126, https://doi.org/10.1109/CVPR.2018.00329 (2018).
Conneau, A., Schwenk, H., Barrault, L. & Lecun, Y. Very deep convolutional networks for text classification. In Lapata, M., Blunsom, P. & Koller, A. (eds.) Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, 1107–1116 (Association for Computational Linguistics, Valencia, Spain, 2017).
S.S. Yadav S.M. Jadhav Deep convolutional neural network based medical image classification for disease diagnosis J. Big Data 6 113 10.1186/s40537-019-0276-2 07964528
Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
Li, Q.et al. Medical image classification with convolutional neural network. In 2014 13th International Conference on Control Automation Robotics and Vision (ICARCV), https://doi.org/10.1109/ICARCV.2014.7064414 (2014).
Dong, C., Loy, C. C. & Tang, X. Accelerating the super-resolution convolutional neural network. In Leibe, B., Matas, J., Sebe, N. & Welling, M. (eds.) Computer Vision – ECCV 2016, 391–407, https://doi.org/10.1007/978-3-319-46475-6_25 (Springer International Publishing, Cham, 2016).
Kim, J., Lee, J. K. & Lee, K. M. Accurate image super-resolution using very deep convolutional networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 1646–1654, https://doi.org/10.1109/CVPR.2016.182 (2016).
L. Yue et al. Image super-resolution: The techniques, applications, and future Signal Process. 128 389 408 2016SigPr.128.389Y 10.1016/j.sigpro.2016.05.002 1094.16004
Z. Wang J. Chen S.C.H. Hoi Deep learning for image super-resolution: A survey IEEE Trans. Pattern Anal. Mach. Intell. 43 3365 3387 10.1109/TPAMI.2020.2982166 32217470 1451.91182
J. Kim C. Jung C. Kim Dual back-projection-based internal learning for blind super-resolution IEEE Signal Process. Lett. 27 1190 1194 2020ISPL..27.1190K 10.1109/LSP.2020.3005043 1446.37101
Wang, X., Xie, L., Dong, C. & Shan, Y. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 1905–1914, https://doi.org/10.1109/ICCVW54120.2021.00217 (2021).
K. Zhang J. Liang L. Van Gool R. Timofte Designing a practical degradation model for deep blind image super-resolution 10.1109/ICCV48922.2021.00475
Wang, Z. et al. Uformer: A general u-shaped transformer for image restoration. https://doi.org/10.1109/CVPR52688.2022.01716 (2022).
Dutta, P., Upadhyay, P., De, M. & Khalkar, R. Medical image analysis using deep convolutional neural networks: Cnn architectures and transfer learning. In 2020 International Conference on Inventive Computation Technologies (ICICT), 175–180. https://doi.org/10.1109/ICICT48043.2020.9112469 (2020).
Alain Horé, D. Z. Image quality metrics: PSNR vs. SSIM. In 20th International Conference on Pattern Recognition, ICPR 2010, Istanbul, Turkey, 23-26 August 2010, 2366–2369. https://doi.org/10.1109/ICPR.2010.579 (IEEE Computer Society, 2010).
A. Mittal A.K. Moorthy A.C. Bovik No-reference image quality assessment in the spatial domain IEEE Trans. Image Process. 21 4695 4708 2012ITIP..21.4695M 3001145 10.1109/TIP.2012.2214050 22910118 1373.94286
Saad, M. A. & Bovik, A. C. Blind quality assessment of videos using a model of natural scene statistics and motion coherency. In 2012 Conference Record of the Forty Sixth Asilomar Conference on Signals, Systems and Computers (ASILOMAR), 332–336, https://doi.org/10.1109/ACSSC.2012.6489018 (2012).
Xiongkuo, M., Huiyu, D., Wei, S., Yucheng, Z. & Guangtao, Z. Perceptual video quality assessment: a survey. Sci. China Inf. Sci67 (2024).
Min, X. et al. Screen content quality assessment: Overview, benchmark, and beyond. ACM Comput. Surv.54, https://doi.org/10.1145/3470970 (2021).
G. Zhai X. Min Perceptual image quality assessment: A survey Sci. China Inf. Sci. 63 10.1007/s11432-019-2757-1 1228.83053 211301
X.P. Burgos-Artizzu et al. Evaluation of deep convolutional neural networks for automatic classification of common maternal fetal ultrasound planes Sci. Rep. 10 10200 2020NatSR.1010200B 1:CAS:528:DC%2BB3cXht1KjurrL 10.1038/s41598-020-67076-5 32576905 7311420 1393.74116
Sendra-Balcells, C. et al. Generalisability of fetal ultrasound deep learning models to low-resource imaging settings in five African countries. Sci. Rep.13 (2023).
A. Sawant S. Kulkarni Ultrasound image enhancement using super resolution Biomed. Eng. Adv. 3 10.1016/j.bea.2022.100039 100039
Shi, W. et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1874–1883. https://doi.org/10.1109/CVPR.2016.207 (2016).
C. Dong C.C. Loy K. He X. Tang Image super-resolution using deep convolutional networks IEEE Trans. Pattern Anal. Mach. Intell. 38 295 307 10.1109/TPAMI.2015.2439281 26761735 0977.60079
A.T. Freitas Data-Driven Approaches in Healthcare: Challenges and Emerging Trends, 65–80 Cham Springer International Publishing 07974880
R.J.G. van Sloun R. Cohen Y.C. Eldar Deep learning in ultrasound imaging Proc. IEEE 108 11 29 10.1109/JPROC.2019.2932116 0969.33001
Y. Zhang Y. Tian Y. Kong B. Zhong Y. Fu Residual dense network for image restoration 10.1109/TPAMI.2020.2968521
Y. Gu et al. Medsrgan: medical images super-resolution using generative adversarial networks Multimedia Tools Appl. 79 21815 21840 10.1007/s11042-020-08980-w
Appati, J. K., Gyamenah, P., Owusu, E. & Yaokumah, W. Deep residual variational autoencoder for image super-resolution. In A. B., Communication and Computing Technology. 91–103, https://doi.org/10.1007/978-3-031-43838-7_7 (Springer Nature Switzerland, Cham, 2023).
Brown, K., Waggener, S., Redfern, A. & Hoyt, K. Faster super-resolution ultrasound imaging with a deep learning model for tissue decluttering and contrast agent localization. Biomed. Phys. Eng. Express 7, https://doi.org/10.1088/2057-1976/ac2f71 (2021).
J.H. Park W. Choi G.Y. Yoon S.J. Lee Deep learning-based super-resolution ultrasound speckle tracking velocimetry Ultrasound Med. Biol. 46 598 609 10.1016/j.ultrasmedbio.2019.12.002 31917044 1254.49019
Y. Shin et al. Context-aware deep learning enables high-efficacy localization of high concentration microbubbles for super-resolution ultrasound localization microscopy Nat. Commun. 15 2932 2024NatCo.15.2932S 1:CAS:528:DC%2BB2cXosVKqsL4%3D 10.1038/s41467-024-47154-2 38575577 10995206 1221.35176
H. Al-Mekhlafi S. Liu Single image super-resolution: A comprehensive review and recent insight Front. Comput. Sci. 18 10.1007/s11704-023-2588-9 07899294 181702
Ledig, C. et al. Photo-realistic single image super-resolution using a generative adversarial network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 105–114, https://doi.org/10.1109/CVPR.2017.19 (2017).
Wang, X.et al. Esrgan: Enhanced super-resolution generative adversarial networks. In Leal-Taixé, L. & Roth, S. (eds.) Computer Vision – ECCV 2018 Workshops, 63–79. https://doi.org/10.1007/978-3-030-11021-5_5 (Springer International Publishing, Cham, 2019).
S. Cammarasana P. Nicolardi G. Patanè Super-resolution of 2d ultrasound images and videos Med. Biol. Eng. Comput. 61 2511 2526 10.1007/s11517-023-02818-x 37195517 10533602 07722585
Lu, Z. et al. Transformer for single image super-resolution. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 456–465. https://doi.org/10.1109/CVPRW56347.2022.00061 (2022).
E. Lu X. Hu Image super-resolution via channel attention and spatial attention Appl. Intell. 52 2260 2268 10.1007/s10489-021-02464-6 1501.90075
Y. Lyu et al. Wsrgan: A wavelet-based gan for super-resolution of plane-wave ultrasound images without sampling loss Eng. Appl. Artif. Intell. 127 10.1016/j.engappai.2023.107384 07920089 107384
F. Dai et al. Df-gam: Cross-domain ultrasound image high-quality reconstruction using a dual frequency-domain guided adaptation model Ultrasound Med. Biol. 50 1403 1414 10.1016/j.ultrasmedbio.2024.05.020 38942620 0464.62069
J. Li Q. Guo S. Peng X. Tan Super-resolution based nodule localization in thyroid ultrasound images through deep learning Curr. Med. Imaging 20 1:CAS:528:DC%2BB2cXhvFeksrbI 10.2174/0115734056269264240408080443 38766836 e15734056269264
O.X. Miguel et al. Deep learning prediction of renal anomalies for prenatal ultrasound diagnosis Sci. Rep. 14 9013 2024NatSR.14.9013M 1:CAS:528:DC%2BB2cXptlShtrw%3D 10.1038/s41598-024-59248-4 38641713 11031588 1404.28008
T. Lehmann C. Gonner K. Spitzer Survey: Interpolation methods in medical image processing IEEE Trans. Med. Imaging 18 1049 1075 1:STN:280:DC%2BD3c7isV2nsg%3D%3D 10.1109/42.816070 10661324 0899.68128
Liang, J. et al. Swinir: Image restoration using swin transformer. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 1833–1844. https://doi.org/10.1109/ICCVW54120.2021.00210 (2021).
Liu, Z. et al. A convnet for the 2020s. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11966–11976. https://doi.org/10.1109/CVPR52688.2022.01167 (2022).
A. Mittal R. Soundararajan A.C. Bovik Making a “completely blind” image quality analyzer IEEE Signal Process. Lett. 20 209 212 2013ISPL..20.209M 10.1109/LSP.2012.2227726 1372.94237
Hossin, M. & M.N, S. A review on evaluation metrics for data classification evaluations. Int. J. Data Min. Knowl. Manag. Process5, 01–11. https://doi.org/10.5121/ijdkp.2015.5201 (2015).
