Explainable AI in the Medical Field: A Survey on Machine Learning Interpretability and Use Cases
Keywords:
XAI , Machine Learning , Black-box, XAI method, metrics, SurveyAbstract
Explainable Artificial Intelligence (XAI) is a branch of Artificial Intelligence (AI) that focuses on developing tools, methodologies, and algorithms capable of delivering interpretable, intuitively understandable insights and rationales for human users. The growing need for XAI in medicine and other fields stems from the increasing demand for equitable and ethical decision-making. It has been established that AI systems largely depends on historical data, therefore, any existing bias or behavior will be perpetuated. As such, deep examination and interpretation are required in this process. Since these black-box models lack transparency and interpretability, several XAI models are developed for the respective domains, ranging from healthcare, military, energy, finance, and industry.The highly sensitive areas, such as healthcare, require knowing the underlying principles of model predictions. Emphasizing feature importance, XAI has improved machine learning models to identify the most critical variables that help improve accuracy and efficiency. With the use of appropriate XAI techniques, actionable insights can be derived that would support informed decisions. In the healthcare sector, the primary objective of XAI is to provide clinicians with tools to effectively evaluate AI-generated data for better patient care. This survey covers Explainable AI, its methods, and their applications in the medical domain.
References
[1] Li, X.H.; Cao, C.C.; Shi, Y.; Bai, W.; Gao, H.; Qiu, L.; and Chen, L.; “A survey of data-driven and knowledge-aware explainable AI”. IEEE Trans. Knowl. Data Eng., 34 (1): 29–49, 2020.
[2] Rudin, C.; “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead”. Nat. Mach. Intell., 1(5): 206–215, 2019.
[3] Guidotti, R.; Monreale, A.; Ruggieri, S.; Turini, F.; Giannotti, F.; and Pedreschi, D.; “A survey of methods for explaining black box models”. ACM Comput. Surv (CSUR), 51(5): 1–42, 2018.
[4] Vellido, A.; "The importance of interpretability and visualization in machine learning for applications in medicine and health care" .Neural Comput. Appl., 32(24): 18069–18083, 2020.
[5] Arrieta, A.B.; Díaz-Rodríguez, N.; Del Ser, J.; Bennetot, A.; Tabik, S.; Barbado, A.; and Herrera, F.; “Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI”. Inf. Fusion, 58: 82–115, 2020.
[6] Adadi, A.; and Berrada, M.; “Peeking inside the black-box: a survey on explainable artificial intelligence (XAI)”. IEEE Access, 6: 52138–52160, 2018.
[7] Zednik, C.; “Solving the black box problem: A normative framework for explainable artificial intelligence”. Philos. Technol., 34(2): 265–288, 2021.
[8] Harris, M.; “The Radical Scope of Tesla's Data Hoard: Every Tesla is providing reams of sensitive data about its driver's life”. IEEE Spectrum, 59 (10): 40-45, 2022.
[9] Gudla, R.; Telidevulapalli, V.S.; Kota, J.S.; and Mandha, G.; "Review on self-driving cars using neural network architectures". World J. Adv. Res. Rev., 16 (2): 736-746, 2022.
[10] Das, S.; Agarwal, N.; Venugopal, D.; Sheldon, F.T.; and Shiva, S.; "Taxonomy and Survey of Interpretable Machine Learning Method." Proceedings of the 2020 IEEE Symposium Series on Computational Intelligence (SSCI), Canberra, Australia, 1-4 December 2020; IEEE: 2020.
[11] Liu, C.Y.J.; and Wilkinson, C.; "Image conditions for machine-based face recognition of juvenile faces". Sci. Justice, 60(1): 43-52, 2020.
[12] Ribeiro, M.T.; Singh, S.; and Guestrin, C.; "'Why Should I Trust You?' Explaining the Predictions of Any Classifier". In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, California, USA, August 13-17 2016; Association for Computing Machinery: New York, USA,1135–1144, 2016.
[13] Alangari, N.; El Bachir Menai, M.; Mathkour, H.; and Almosallam, I.; “Exploring evaluation methods for interpretable machine learning: A survey”. Information, 14 (8): 469, 2023.
[14] Richardson, R.; Schultz, J.M.; and Crawford, K.; “Dirty data, bad predictions: How civil rights violations impact police data, predictive policing systems, and justice”. NYUL Rev. 94: 15, 2019.
[15] Srinivasu, P.N.; Sandhya, N.; Jhaveri, R.H.; and Raut, R.; “From blackbox to explainable AI in healthcare: existing tools and case studies” .Mob. Inf. Syst., 2022(1): 8167821, 2022.
[16] Azodi, C.B.; Tang, J.; and Shiu, S.H.; “Opening the black box: interpretable machine learning for geneticists”. Trends Genet., 36(6): 442–455, 2020.
[17] Borys, K.; Schmitt, Y.A.; Nauta, M.; Seifert, C.; and Krämer, N.; Friedrich, C.M.; Nensa, F.; “Explainable AI in medical imaging: An overview for clinical practitioners–saliency-based XAI approaches,” Eur. J. Radiol., 162: 110787, 2023.
[18] Cao, Q.H.; Nguyen, T.T.H.; Nguyen, V.T.K.; and Nguyen, X.P.; “A Novel Explainable Artificial Intelligence Model in Image Classification Problem”.arXiv Prepr., 2023.
[19] Lundberg, S.M.; and Lee, S.I; "A Unified Approach to Interpreting Model Predictions". Proceedings of the 31st International Conference on Neural Information Processing Systems, California, USA, 2017; Curran Associates Inc.: NY, USA, 2017.
[20] Nicodeme, C.; "Build Confidence and Acceptance of AI-Based Decision Support Systems – Explainable and Liable AI". Proceedings of the 2020 13th International Conference on Human System Interaction (HSI), Tokyo, Japan, 6–8 July 2020; IEEE: Piscataway, USA, 2020.
[21] Gunning, D.; and Aha, D.; “DARPA’s Explainable Artificial Intelligence (XAI) Program”. AI Mag., 40(2): 44–58, 2019.
[22] Linardatos, P.; Papastefanopoulos, V.; and Kotsiantis, S.; “Explainable AI: A review of machine learning interpretability methods”. Entropy, 23(1): 18, 2020.
[23] Aslam, N.; Khan, I.U.; Mirza, S.; AlOwayed, A.; Anis, F.M.; Aljuaid, R.M.; and Baageel, R.; “Interpretable machine learning models for malicious domains detection using explainable artificial intelligence (XAI)”. Sustainability, 14(12): 7375, 2022.
[24] Holzinger, A.; Saranti, A.; Molnar, C.; Biecek, P.; and Samek, W.; "Explainable AI Methods - A Brief Overview". In xxAI - Beyond Explainable AI, Lecture Notes in Computer Science; Holzinger, A.; Goebel, R.; Fong, R.; Moon, T.; Müller, K.R.; Samek, W., Eds.; Springer: Cham, Switzerland, 13–38, 2022.
[25] Calegari, R.; Ciatto, G.; Dellaluce, J.; and Omicini, A.; "Interpretable Narrative Explanation for ML Predictors with LP: A Case Study for XAI". Proceedings of the 20th Workshop "From Objects to Agents", Parma, Italy, 26th–28th June 2019; Bergenti, F., Monica, S.; CEUR-WS.org: Pisa, Italy, 2019.
[26] Linardatos, P.; Papastefanopoulos, V.; and Kotsiantis, S.; “Explainable AI: A review of machine learning interpretability methods” .Entropy, 23(1), 18, 2020.
[27] Miller, T.; “Explanation in artificial intelligence: Insights from the social sciences” .Artif. Intell., 267: 1–38, 2019.
[28] Kim, B.; and Doshi-Velez, F.; “Machine learning techniques for accountability”. AI Magazine, 42(1): 47–52, 2021.
[29] Murdoch, W.J.; Singh, C.; Kumbier, K.; Abbasi-Asl, R.; and Yu, B.; “Definitions, methods, and applications in interpretable machine learning”. Proc. Natl. Acad. Sci., 116(44): 22071–22080, 2019.
[30] Murdoch, W.J.; Singh, C.; Kumbier, K.; Abbasi-Asl, R.; and Yu, B.; "Definitions, methods, and applications in interpretable machine learning". Proc. Natl. Acad. Sci., 116 (44): 22071-22080, 2019.
[31] Gilpin, L. H.; Bau, D.; Yuan, B. Z.; Bajwa, A.; Specter, M.; and Kagal, L.; "Explaining Explanations: An Overview of Interpretability of Machine Learning". Proceedings of the 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), Turin, Italy, 2018; IEEE: Turin, Italy, 2018.
[32] Gacto, M.J.; Alcalá, R.; and Herrera, F.; “Interpretability of linguistic fuzzy rule-based systems: An overview of interpretability measures”. Inf. Sci., 181(20): 4340–4360, 2011.
[33] He, C.; Ma, M.; and Wang, P.; “Extract interpretability-accuracy balanced rules from artificial neural networks: A review” .Neurocomputing, 387: 346–358, 2020.
[34] Loi, M.; Ferrario, A.; and Viganò, E.; “Transparency as design publicity: explaining and justifying inscrutable algorithms”. Ethics Inf. Technol., 23(3): 253–263, 2021.
[35] Sachan, S.; Almaghrabi, F.; Yang, J.B.; and Xu, D.L.; “Evidential reasoning for preprocessing uncertain categorical data for trustworthy decisions: An application on healthcare and finance”. Expert Syst. Appl., 185: 115597, 2021.
[36] Hamon, R.; Junklewitz, H.; and Sanchez, I.; "Robustness and Explainability of Artificial Intelligence". Publications Office of the European Union, Luxembourg, 2020.
[37] Bhatt, U.; Xiang, A.; Sharma, S.; Weller, A.; Taly, A.; Jia, Y.; and Eckersley, P.; “Explainable machine learning in deployment”. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 648–657, 2020.
[38] Belle, V.; and Papantonis, I.; “Principles and practice of explainable machine learning” .Front. Big Data, 4: 688969, 2021.
[39] Carvalho, D.V.; Pereira, E.M.; and Cardoso, J.S.; “Machine learning interpretability: A survey on methods and metrics”. Electronics, 8(8): 832, 2019.
[40] Guleria, P.; and Sood, M.; “Explainable AI and machine learning: performance evaluation and explainability of classifiers on educational data mining inspired career counseling”. Educ. Inf. Technol., 28(1): 1081–1116, 2023.
[41] Wen, Y.; and Holweg, M.; “A Phenomenological Perspective on AI Ethical Failures: The Case of Facial Recognition Technology”. AI & Society, 39(4), 1929–1946, 2024.
[42] Guidotti, R.; Monreale, A.; Giannotti, F.; Pedreschi, D.; Ruggieri, S.; and Turini, F.; “Factual and counterfactual explanations for black box decision making”. IEEE Intell. Syst., 34(6): 14–23, 2019.
[43] Panigutti, C.; Guidotti, R.; Monreale, A.; and Pedreschi, D.; "Explaining Multi-label Black-Box Classifiers for Health Applications". In Precision Health and Medicine; Shaban-Nejad, A., Michalowski, M., Eds.; Springer: Cham, Switzerland, 155-167, 2020.
[44] Hassija, V.; Chamola, V.; Mahapatra, A.; Singal, A.; Goel, D.; Huang, K.; and Hussain, A.; "Interpreting black-box models: a review on explainable artificial intelligence". Cogn. Comput., 16 (1): 45–74, 2024.
[45] Deng, H.; “Interpreting tree ensembles with intrees”. Int. J. Data Sci. Anal., 7(4): 277–287, 2019.
[46] Guleria, P.; Naga Srinivasu, P.; Ahmed, S.; Almusallam, N.; and Alarfaj, F.K.; “XAI framework for cardiovascular disease prediction using classification techniques”. Electronics, 11(24): 4086, 2022.
[47] Singh, A.; Sengupta, S.; and Lakshminarayanan, V.; "Explainable deep learning models in medical image analysis". J. Imaging, 6 (6): 52, 2020.
[48] Tukey, J.W.; “Exploratory Data Analysis”. Reading/Addison-Wesley, 2: 131-160, 1977.
[49] Jiarpakdee, J.; Tantithamthavorn, C.K.; Dam, H.K.; and Grundy, J.; "An empirical study of model-agnostic techniques for defect prediction models". IEEE Trans. Softw. Eng., 48 (1): 166–185, 2020.
[50] Yan, F.; Wen, S.; Nepal, S.; Paris, C.; and Xiang, Y.; "Explainable machine learning in cybersecurity: A survey". Int. J. Intell. Syst., 37 (12): 12305–12334, 2022.
[51] Wang, N.; Zhang, H.; Dahal, A.; Cheng, W.; Zhao, M.; and Lombardo, L.; "On the use of explainable AI for susceptibility modeling: Examining the spatial pattern of SHAP values". Geosci. Front., 15(4): 101800, 2024.
[52] Kök, I.; Okay, F.Y.; Muyanlı, Ö.; and Özdemir, S.; "Explainable artificial intelligence (XAI) for Internet of Things: a survey". IEEE Internet Things J., 10 (16): 14764–14779, 2023
[53] Roy, T.; Das, P.; Jagirdar, R.; Shhabat, M.; Abdullah, M.S.; Kashem, A.; and Rahman, R.; "Prediction of mechanical properties of eco-friendly concrete using machine learning algorithms and partial dependence plot analysis". Smart Constr. Sustain. Cities, 3 (1): 2, 2025.
[54] Friedman, J.H.; “Greedy function approximation: a gradient boosting machine” .Ann. Statist., 29: 1189–1232, 2001.
[55] Goldstein, A.; Kapelner, A.; Bleich, J.; and Pitkin, E.; “Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation”. J. Comput. Graph. Stat., 24(1): 44–65, 2015.
[56] Hassija, V.; Chamola, V.; Mahapatra, A.; Singal, A.; Goel, D.; Huang, K.; and Hussain, A.; “Interpreting black-box models: a review on explainable artificial intelligence”. Cogn. Comput., 16(1): 45-74, 2024.
[57] Apley, D.W.; and Zhu, J.; “Visualizing the effects of predictor variables in black box supervised learning models”. J. R. Stat. Soc. Ser. B: Stat. Methodol., 82(4): 1059–1086, 2020.
[58] Dwivedi, R.; Dave, D.; Naik, H.; Singhal, S.; Omer, R.; Patel, P.; and Ranjan, R.; "Explainable AI (XAI): Core ideas, techniques, and solutions". ACM Comput. Surv., 55 (9): 1–33, 2023.
[59] García, M.V.; and Aznarte, J.L.; “Shapley additive explanations for NO2 forecasting”. Ecol. Informatics, 56: 101039, 2020.
[60] Keerthana, C.S.; Nalluri, S.C.; Muskaan, S.; and Sadagopan, P.; "Explainable AI in Credit Card Fraud Detection: SHAP and LIME for Machine Learning Models".Proceedings of the 2025 10th International Conference on Signal Processing and Communication (ICSC), Noida, India, 2025; Publisher: 2025.
[61] Lundberg, S.M.; and Lee, S.-I.; "A Unified Approach to Interpreting Model Predictions". Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, 2017; Curran Associates Inc.: Red Hook, USA, 2017.
[62] Mothilal, R.K.; Sharma, A.; and Tan, C.; "Explaining Machine Learning Classifiers Through Diverse Counterfactual Explanations". 2020 Conference on Fairness, Accountability, and Transparency; Association for Computing Machinery: New York, USA, 2020.
[63] Salih, A.M.; Raisi‐Estabragh, Z.; Boscolo Galazzo, I.; Radeva, P.; Petersen, S.E.; Lekadir, K.; and Menegaz, G.; “A perspective on explainable artificial intelligence methods: SHAP and LIME”. Adv. Intell. Syst., 7(1): 2400304, 2025.
[64] Zhao, C.; Liu, J.; and Parilina, E.; "ShapG: New Feature Importance Method Based on the Shapley Value". Eng. Appl. Artif. Intell., 148(1): 110409, 2025.
[65] Holzinger, A.; Langs, G.; Denk, H.; Zatloukal, K.; and Müller, H.; "Causability and Explainability of Artificial Intelligence in Medicine". Wiley Interdiscip. Rev. Data Min. Knowl. Discov., 9(4): 1312, 2019.
[66] Amin, A.; Hasan, K.; Zein-Sabatto, S.; Chimba, D.; Ahmed, I.; and Islam, T.; “An explainable AI framework for artificial intelligence of medical things”. In 2023 IEEE Globecom Workshops (GC Wkshps), 2097–2102, 2023.
[67] Santafe, G.; Inza, I.; and Lozano, J.A.; “Dealing with the evaluation of supervised classification algorithms”. Artif. Intell. Rev., 44: 467-508, 2015.
[68] Canbek, G.; Taskaya Temizel, T.; and Sagiroglu, S.; "PToPI: A comprehensive review, analysis, and knowledge representation of binary classification performance measures/metrics". SN Comput. Sci., 4 (1): 13, 2022.
[69] Zhu, W.; Zeng, N.F.; and Wang, N.; "Sensitivity, Specificity, Accuracy, Associated Confidence Interval and ROC Analysis with Practical SAS". Proceedings of the NESUG Conference: Health Care and Life Sciences, New York, USA, 2010; NESUG: 2010.
[70] Tharwat, A.; “Classification assessment methods”. Appl. Comput. Informat., 17(1): 168-192, 2021.
[71] Dembczynski, K.; Waegeman, W.; Cheng, W.; and Hüllermeier, E.; "An Exact Algorithm for F-Measure Maximization". In: Adv. Neural Inf. Process. Syst., Proceedings of the 24th Annual Conference on Neural Information Processing Systems (NeurIPS), Granada, Spain, 2011; Curran Associates: NY, USA, 2011.
[72] Hicks, S. A.; Strümke, I.; Thambawita, V.; Hammou, M.; Riegler, M. A.; Halvorsen, P.; and Parasa, S.; “On evaluation metrics for medical applications of artificial intelligence”. Sci. Rep., 12(1): 5979, 2022.
[73] Kumar, R.; and Indrayan, A.; “Receiver operating characteristic (ROC) curve for medical researchers”. Indian Pediatr., 48: 277–287, 2011.
[74] Haitsam, H.; "Human Detection With Plant Bioelectric Signal". J. Qua Teknika, 15(01): 35-42, 2025.
[75] Pepe, M. S.; “Receiver operating characteristic methodology”. J. Am. Stat. Assoc., 95: 308–311, 2000.
[76] Walter, S. D.; “The partial area under the summary ROC curve”. Stat. Med., 24(13): 2025-2040, 2005.
[77] Marzban, C.; “The ROC curve and the area under it as performance measures”. Weather Forecast., 19(6): 1106–1114, 2004.
[78] Faraggi, D.; and Reiser, B.; “Estimation of the area under the ROC curve”. Stat. Med., 21(20): 3093-3106, 2002.
[79] Carrington, A.M.; Manuel, D.G.; Fieguth, P.W.; Ramsay, T.; Osmani, V.; Wernly, B.; and Holzinger, A.; "Deep ROC Analysis and AUC as Balanced Average Accuracy, for Improved Classifier Selection, Audit and Explanation". IEEE Trans. Pattern Anal. Mach. Intell., 45(1): 329-341, 2022.
[80] Sheu, R.K.; and Pardeshi, M.S.; "A Survey on Medical Explainable AI (XAI): Recent Progress, Explainability Approach, Human Interaction and Scoring System". Sensors, 22(20): 8068, 2022.
[81] Porto, R.; Molina, J. M.; Berlanga, A.; and Patricio, M. A.; “Minimum relevant features to obtain explainable systems for predicting cardiovascular disease using the statlog data set”. Appl. Sci., 11(3), 1285, 2021.
[82] Moreno-Sanchez, P. A.; "Development of an Explainable Prediction Model of Heart Failure Survival by Using Ensemble Trees". Proceedings of the 2020 IEEE International Conference on Big Data (Big Data), Atlanta, USA, 2020; IEEE: 2020.
[83] Peng, J.; Zou, K.; Zhou, M.; Teng, Y.; Zhu, X.; and Zhang, F.; and Xu, J.; “An explainable artificial intelligence framework for the deterioration risk prediction of hepatitis patients”. J. Med. Syst., 45(5): 61, 2021.
[84] Ghosh, S. K.; and Khandoker, A. H.; “Investigation on explainable machine learning models to predict chronic kidney diseases”. Sci. Rep., 14(1): 3687, 2024.
[85] Salekin, A.; and Stankovic, J.; "Detection of Chronic Kidney Disease and Selecting Important Predictive Attributes". Proceedings of the 2016 IEEE International Conference on Healthcare Informatics (ICHI), Chicago, USA, 2016; IEEE: 2016.
[86] Parmar, C.; Grossmann, P.; Rietveld, D.; Rietbergen, M. M.; Lambin, P.; and Aerts, H. J.; “Radiomic machine-learning classifiers for prognostic biomarkers of head and neck cancer”. Front. Oncol., 5: 272, 2015.
[87] Macyszyn, L.; Akbari, H.; Pisapia, J. M.; Da, X.; Attiah, M.; Pigrish, V.; and Davatzikos, C.; “Imaging patterns predict patient survival and molecular subtype in glioblastoma via machine learning techniques”. Neuro-Oncol., 18(3): 417–425, 2015.
[88] Tabari, A.; D’Amore, B.; Cox, M.; Brito, S.; Gee, M. S.; Wehrenberg-Klee, E.; and Daye, D.; “Machine Learning-Based Radiomic Features on Pre-Ablation MRI as Predictors of Pathologic Response in Patients with Hepatocellular Carcinoma Who Underwent Hepatic Transplant”. Cancers, 15(7): 2058, 2023.
[89] Severn, C.; Suresh, K.; Görg, C.; Choi, Y. S.; Jain, R.; and Ghosh, D.; “A Pipeline for the Implementation and Visualization of Explainable Machine Learning for Medical Imaging Using Radiomics Features”. Sensors, 22(14): 5205, 2022.
[90] Xie, Y.; Li, X.; Yang, S.; Jia, F.; Han, Y.; and Huang, M.; "Radiomics Models Using Machine Learning Algorithms to Differentiate the Primary Focus of Brain Metastasis". Transl. Cancer Res., 14(2): 731, 2025.
[91] Barda, A.J.; "Design and Evaluation of User-Centered Explanations for Machine Learning Model Predictions in Healthcare". University of Pittsburgh: Pittsburgh, USA, 2020.
[92] Yoo, T. K.; Ryu, I. H.; Choi, H.; Kim, J. K.; Lee, I. S.; Kim, J. S.; Lee, G.; and Rim, T. H.; “Explainable Machine Learning Approach as a Tool to Understand Factors Used to Select the Refractive Surgery Technique on the Expert Level”. Transl. Vis. Sci. Technol., 9(2): 8, 2020.
[93] Cheng, F.; Liu, D.; Du, F.; Lin, Y.; Zytek, A.; Li, H.; Qu, H.; and Veeramachaneni, K.; “Vbridge: Connecting the Dots Between Features and Data to Explain Healthcare Models”. IEEE Trans. Vis. Comput. Graph., 28(1): 378–388, 2021.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Mahmood Thamer, Zainab N. Sultani
This work is licensed under a Creative Commons Attribution 4.0 International License.
.jpg)
