The incidence of in-hospital cardiac arrest (IHCA) is about 8.5 records for every 1000 admissions [1]. For IHCA patients, the rate of survival to hospital discharge is about 39.5%, and only 28.3% of IHCA patients regain independent or partially independent lives [2]. Previous studies suggest that detecting adverse signs and symptoms early and adjusting medical care accordingly has the potential to improve a patient’s prognosis by properly allocating healthcare resources and reducing future healthcare needs [3]. Machine learning methods, especially deep learning approaches, have been shown to be more effective than traditional epidemiological studies at uncovering disease patterns and understanding patient disease trajectories [4, 5, 6, 7, 8]. However, since prior selection of potential risk factors is required in epidemiological research methods, these approaches are time consuming and prone to bias if conducted manually. Although machine-learning approaches provide promising levels of prediction accuracy, their lack of interpretability has limited their adoption in a clinical setting. It is important to develop a robust and trustworthy framework consisting of interpretable methods that can explain why a certain prediction was made for a given case [9, 10, 11]. However, there are relatively few studies regarding explainable deep learning models for cardiac arrest prediction. Some researchers also advocate for a careful and thorough validation of these approaches [12], which has not yet been undertaken.

In this study, we used a pre-trained Hierarchical Vectorizer (HVec) deep learning model to predict the mortality of cardiac arrest patients using data from Taiwan’s large-scale National Health Insurance Research Database (NHIRD). This model achieved a 0.711 area under the receiver operating characteristic (AUROC) score when predicting patients’ 30-day mortality after each clinical record and a 0.808 AUROC score when predicting patients’ 30-day mortality after IHCA [13]. Based on this, we introduced a deep learning interpretation Deep SHapley Additive exPlanations (D-SHAP) framework to determine the correlation between input features and 30-day mortality probability of IHCA patients. In clinical settings, the diagnosis code is a key feature used by physicians to estimate patient health status. The diagnosis code input feature was used to check the performance of the D-SHAP framework. A linear combination method is proposed to aggregate the SHAP values and thus generate the impact of the diagnosis code on the mortality probability [14, 15]. The physicians’ opinion was introduced as the benchmark to measure the similarity between the impact calculated from the D-SHAP framework and human experts’ analyses.

In this study, we aim to evaluate that D-SHAP can capture the diagnosis code with the highest mortality risk from a deep neural network and generate a result consistent with the physician’s diagnosis.

This study was approved by the Institutional Review Board of National Taiwan University Medical College.

2.3 Deep SHapley Additive exPlanations (D-SHAP) Deep Learning Models

Deep SHapley Additive exPlanations (D-SHAP) is a method that can provide deep learning model explanation using linear approximation and derivative chain rule for each input and output, referred to as local input/output [17]. The methodology and calculation details were provided in the Supplementary Material for reference and further examination. The explainable deep learning model will provide each clinical record a continuous D-SHAP value for predicting the probability of 30-day mortality.

For the diagnosis codes in each record, current and historical D-SHAP impacts are analyzed from the model’s perspective to determine the importance of each diagnosis code (Fig. 1). The current D-SHAP value is defined by checking the diagnosis codes of the current event, providing a scale to measure the likelihood of 30-day mortality for the individual. The historical D-SHAP value is defined by checking the diagnosis codes of all previous events and provides a scale to measure the likelihood of 30-day mortality. In this experiment, the high/low SHAP impact segmentation criterion is set according to the results. In principle, records with SHAP prediction value $\ge$0.25 are considered as high SHAP impact, and these diagnosis codes can predict 30-day mortality. In contrast, records with SHAP prediction value $\le$0.10 are considered as low SHAP impact; these diagnosis codes can negatively predict 30-day mortality.

Fig. 1.

Aggregating historical diagnosis SHapley Additive exPlanations (SHAP) versus current diagnosis SHAP.

In this study, the SHAP value is validated against a physician’s decision to determine the consistency between the SHAP value and human knowledge.

In this paper, we proposed a D-SHAP machine learning model that can be used to explain deep neural network modeling. The electronic health records of an IHCA cohort were investigated using our D-SHAP framework to find the most important diagnosis codes leading to mortality. After comparison with physicians’ annotations, we found that most of the important diagnosis codes that could lead to mortality can be captured by our D-SHAP framework. One of the diagnoses, urinary tract infection, showed a high discrepancy between our D-SHAP model and clinical judgment. Urinary tract infection is a relatively common disease leading to admission, especially in seniors or patients with multiple comorbidities [18]. We assume that the high prevalence of urinary tract infection in our dataset with its high frequency of comorbidities with dangerous diagnoses including pneumonia, acute respiratory failure, and sepsis might mislead the machine learning process. Results show that our framework can determine some vital diagnosis codes that cannot be found by conventional clinical judgment. However, physicians should always carefully evaluate the results of machine learning and consider underlying pathophysiological mechanisms.

Along with the recent explosive development of machine learning in medicine, several arguments about its utility in clinical practice have manifested, especially regarding black-box and overfitting issues [19, 20, 21, 22, 23]. With improvements in computer science, explainable machine learning models have been widely used recently to address the drawbacks of traditional machine learning models; they have been used in several areas of medicine [5, 24, 25, 26, 27, 28, 29], and they also provide prediction algorithms for use by clinical physicians [30]. These studies proposed several models with high predictive values for critical illness. They also proposed several predictive factors using explainable deep learning models such as SHAP and locally interpretable model-agnostic explanations (LIME), yielding insight into the mechanisms of these models. However, how reasonable these generated factors are is still in question. In addition to post-hoc judgment based on clinical rationales, which carry the risk of confirmation bias, further double-blind studies are needed for more rigorous validation [12].

In this study, not only did we propose a deep learning interpretation framework for predicting mortality by EHRs of NHIRD, but we also performed a prospective validation against the judgment of clinical physicians. To the best of our knowledge, this is the first study using a prospective method to validate an explainable deep learning model. We used diagnosis for the index as an important feature of EMRs that covers patients’ overall status as well as physicians’ judgment. To correspond with D-SHAP values, we innovated a 1–4 score for each visit by clinical judgment. In this experiment, the prevalence of each diagnosis is a key issue. Some diagnoses seldom appeared and had a very small sample size, so we cannot solely calculate the mean score of each diagnosis. Also, some diagnoses are strong predictive factors for 30-day mortality while other diagnoses are strong protective factors. Therefore, we propose measuring the importance of a diagnosis by calculating the difference in probability between high and low scores. However, the prevalence of diagnoses within the dataset was still a major confounding factor. In addition to those diagnoses with higher risk, those with higher prevalence will also have higher ranking. For example, cardiac arrest should be a stronger predictor of mortality than any other. However, due to the relatively low frequency of cardiac arrest featuring as the diagnosis, the importance of this diagnosis scored lower than other more common diseases such as pneumonia or respiratory failure. Also, frequently occurring diseases such as urinary tract infections are expected to have higher rankings, especially those with more co-prevalence with other severe comorbidities. Therefore, the ranking in our study did not emphasize the order of severity but only indicated those diagnoses that should bear greater consideration. This experiment also illustrated the point that the end users of machine learning models should always carefully evaluate the results and consider the structure of the original database.

This study has several implications. We found that an explainable deep learning model can determine diagnosis with clinical significance for a complicated database such as NHIRD. This model can be utilized as an early warning system for patients who are at risk of mortality according to recent EHRs. Patients with a high risk of mortality could be identified and re-evaluated at each clinical visit. With an explainable deep learning model, several diagnoses or risk factors can be proposed for helping physicians to make the most effective clinical decisions. We consider the present study as a preliminary study for future work and demonstrate that our model can be an effective tool with reasonable explainability. In Taiwan, the NHI database contains over 99% of the population’s medical information for insurance purposes. In the future, we hope to establish an alarm system based on NHIRD by connecting hospital EHRs and deep learning software within NHIRD, which can be universally applied to Taiwan’s population for predicting severe, high-risk medical conditions such as cardiac arrest [31, 32, 33, 34, 35].

This study had several limitations. First, the IHCA cohort was retrospectively collected using NHIRD. These patients were usually diagnosed with critical illnesses and multiple comorbidities during the study period. The implications of extending this model to the general population or other datasets are unknown. Second, only the diagnosis code was used in this study due to study design and the complexity of NHIRD. Explainability of the whole model was not evaluated or validated by this study. Third, as mentioned above, the prevalence of each diagnosis would have an impact on its calculated importance. Since NHIRD is used for reimbursement purposes, diagnoses other than primary diagnosis for admission were not always recorded by physicians. The gap between NHIRD records and clinical diagnosis should be considered. Fourth, the calculation formula for the importance of each diagnosis was designed solely for our validation experiment. Due to the lack of similar studies in the literature, the methodology used in this study should be applied with caution and further validation is needed. Finally, further studies are needed to evaluate the utility of explainable deep learning models in real-world medical applications and thus determine whether this system can improve patients’ outcomes.

In this study, the D-SHAP framework was found to be an effective tool for explaining deep neural networks in the prediction of patients’ 30-day mortality. Most of the important diagnosis codes that could lead to mortality, including respiratory failure, sepsis, pneumonia, shock, and acute kidney injury, can be captured by our D-SHAP framework. However, physicians should always carefully evaluate the results of machine learning, taking into account underlying pathophysiological mechanisms.

The data that support the findings of this study are available from NHIRD but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of NHIRD.

HMD, AW, SA, and CHH contributed to study concept and design; CYC, YPC, LWW, PIS, WSL, and MST were contributed to the acquisition of data; HMD, AW, and SA analyzed the data; CYC, SA and CHH interpreted the data; the first draft of the article was prepared by CYC, HMD, and AW; SA, YPC, LWW, PIS, WSL, and MST were involved in critical revision of the content; the final revision was made by CYC and CHH. All authors read and approved the final manuscript. All authors have participated sufficiently in the work and agreed to be accountable for all aspects of the work.

This study was approved by the Institutional Review Board of National Taiwan University Medical College, approval number: 201806057RIPC. Patient’s informed consent is waived.

Not applicable.

This study was supported by the Ministry of Science and Technology, Taiwan (project: 109-2634-F-002-031).

Hadi Moghadas-Dastjerdi and Adrian Winkler are affiliated with Knowtions Research Inc. Shuang Ao is a former employee of the same company. All authors and Knowtions Research Inc. confirm no conflicts of interest. Chien-Hua Huang is serving as Guest Editor of this journal. We declare that Chien-Hua Huang had no involvement in the peer review of this article and has no access to information regarding its peer review. Full responsibility for the editorial process for this article was delegated to Giuseppe Boriani.