Tech SpaghettiDemystifying the Black Box: A Hands-On Guide to Explainable AI (XAI)
TL;DR
🧠 Explainability is Trust – High-performance models (like XGBoost) are often “black boxes”. Explainable AI (XAI) peels back the layers to reveal how and why they make decisions.
🚀 Streamlit in action – We build and explain an interactive heart disease classifier, showing XAI in a clinical context using an interactive live application.
🧩 Feature importance (ELI5) – Permutation Importance identifies which patient attributes (e.g. major vessels, cholesterol) carry the most predictive weight.
📉 Partial Dependence (PDP) – Visualises the global relationship between specific inputs (like max heart rate) and heart disease risk.
🔬 SHAP (Game Theory) – Provides both hyper-local explanations (why patient #1 is high-risk) and birds-eye global insights using coalition game theory.
Introduction
We are living in an era where Machine Learning models are making increasingly critical decisions. From approving credit cards to diagnosing life-threatening conditions, AI is driving automation at an unprecedented scale. But as models become more powerful like XGBoost, Deep Neural Networks, and Large Language Models; they also become more complex. The underlying algorithms acts like “black boxes”. Black boxes are systems where we feed inputs and receive highly accurate predictions, but have absolutely no visibility into the logic under the hood.
In low-stakes scenarios (like movie recommendations), this opacity is acceptable. In high-stakes fields like healthcare and self driving cars, it is dangerous and could potential have real-world implications. A doctor cannot simply trust an AI that says, “This patient has a 90% chance of heart disease,” without asking why.
This is where Explainable AI (XAI) steps in. XAI is the collection of tools and frameworks that make machine learning models transparent, interpretable, and accountable.
To see this in action, we built an interactive Heart Disease Explainable AI Streamlit App.
Let’s dive into how we can use three core pillars of explainability to dissect an XGBoost classifier trained on Cleveland heart disease data.
If algorithms meet explainability requirements, they provide a basis for justifying decisions, tracking and thereby verifying them, improving the algorithms, and exploring new facts.
The Core Engine: XGBoost on Heart Disease Data
Our application uses the popular Kaggle Heart Disease dataset, which contains 14 patient attributes, including age, sex, chest pain type, max heart rate, and the number of major vessels.
We trained an XGBoost Classifier on this dataset. XGBoost is a decision-tree-based ensemble algorithm known for high accuracy but notorious for its non-linear, complex decision boundaries.
Once trained, the model achieves strong classification metrics on validation data. But to transition this model from a research experiment into a trustworthy clinical tool, we must explain its predictions. We do this using three distinct techniques.
Pillar 1: Permutation Importance (What does the model value?)
Before we look at individual patients, we want to know: what features does the model care about most across the entire dataset?
One of the fastest and most intuitive ways to calculate this is Permutation Importance (rendered in our app via the eli5 library).
How it works
from eli5.sklearn import PermutationImportance # Fit Permutation Importance on test data perm = PermutationImportance(model, random_state=1).fit(X_test, y_test) In our heart prediction model, the top 3 most important features identified are number_of_major_vessels, cholestrol, and st_depression_rt_rest.
This aligns perfectly with clinical literature: the number of clear major blood vessels and high cholesterol levels are leading physiological indicators of cardiovascular health.
Pillar 2: Partial Dependence Plots (How do features affect predictions?)
While Permutation Importance tells us which features are important, it doesn’t tell us the direction of the relationship. Does a higher heart rate increase or decrease the risk of heart disease?
To answer this, we use Partial Dependence Plots (PDP), implemented using scikit-learn’s Partial Dependence Display.
How it works
PDPs isolate a single feature and show its marginal effect on the predicted outcome. The algorithm takes the dataset, forces the target feature to a specific value for all rows (e.g., setting every patient’s max_heart_rate to 140), and calculates the average predicted probability. It repeats this across a grid of values.
In our Streamlit app, we render these plots dynamically using Matplotlib:
from sklearn.inspection import PartialDependenceDisplay import matplotlib.pyplot as plt fig, ax = plt.subplots(figsize=(6, 4), dpi=100) pdp_iso = PartialDependenceDisplay.from_estimator( model, X_test, features=[feature_index], feature_names=list(features_list), ax=ax ) Clinical Insights from PDP:
- max_heart_rate: As the patient’s maximum heart rate increases, the PDP line rises, indicating a clear increase in heart disease probability.
- number_of_major_vessels: Conversely, as the number of clear major vessels increases from 0 to 3, the risk curve drops sharply. More clear vessels mean better blood flow, reducing cardiovascular risk.
Pillar 3: SHAP Values (Local & Global Explanations)
If a doctor is sitting with a specific patient, general dataset-level statistics aren’t enough. They need to know: why did the AI predict that this specific patient has a high risk of heart disease?
For local, patient-level explanations, we use SHAP (SHapley Additive exPlanations).
Based on coalition game theory, SHAP treats each feature value of a patient as a “player” in a game, where the model’s prediction is the payout. It calculates the fair contribution of each feature to the difference between the actual prediction and the average baseline prediction.
import shap explainer = shap.TreeExplainer(model) shap_values = explainer.shap_values(patient_data) Local Interpretability (Individual Patients)
In our app, you can select individual patients and see their Force Plots:
- Features that push the risk higher are shown in red/pink.
- Features that pull the risk lower are shown in blue.
For example, for a high-risk patient, SHAP highlights that being a male (sex_male = 1) and having a thalassemia defect (thalassemia_reversible_defect = 1) are the dominant drivers pushing their score above the baseline.
Global Interpretability (Summary Plot)
SHAP also provides a Summary Plot that combines local explanations across all patients. Each point on the plot represents a single patient:
- Y-axis: Features ordered by overall impact.
- X-axis: SHAP value (impact on model output).
- Color: High feature values (red) vs. low feature values (blue).
This single chart instantly reveals that high values of Thalassemia defects (red dots on the right side) push predictions higher, while having more major vessels (red dots on the left side) pulls predictions down.
Closing Thoughts: Trust is the New Metric
Subscribe to keep reading
It's a subscribers only post. Subscribe to get access to the rest of this post and other subscriber-only content.
Subscribe #AI #AIML #Data #DeepLearning #ELI5 #Explainability #GameTheory #PartialDependencyPlot #PDP #Python #ResponsibleAI #SHAP #Trust #XAI #XGBoost
DSigmund
sen
Stylus
Adhidarma Hadiwinoto 
Goose