◆ MACHINE LEARNING MODELING · CLINICAL CARDIOLOGY ◆

Heart Disease Prediction

Advanced Machine Learning Pipeline · Baseline Model Upgrades · Stacking Ensembles

🤖 Classification 🛡️ GridSearchCV ⚙️ Pipeline Security 🧬 Stacking Ensemble

Dataset

UCI Heart Disease

Base Models

DT, RF, KNN, SVM, XGB

Validation

Stratified 5-Fold CV

Ensembles

Voting & Stacking

Early detection of cardiovascular disease is critical for clinical decision-making. Standard machine learning workflows often suffer from data leakage (e.g. scaling before train-test split) or fake order representation (using label encoding on nominal categories).

Modeling Methodology & Safeguards:

• Preprocessing Pipeline: Volumetric and scaling steps are bundled inside a Scikit-Learn ColumnTransformer to prevent data leakage.
• Power Transforms: Continuous features with high skewness are normalized using the Yeo-Johnson power transform to meet Gaussian assumptions for linear models.
• VIF Multicollinearity Check: Evaluating Variance Inflation Factors to audit linear dependence among predictors.
• Model Comparison: Training Decision Trees, Random Forests, KNN, SVM, and XGBoost models, tuning hyper-parameters using GridSearchCV with stratified 5-fold cross-validation.
• Ensemble Stacking: Meta-modeling combining base predictions under a final logistic regression estimator to maximize out-of-sample F1 and accuracy.

01

Setup & Visual Settings

Setup complete | High-Contrast Clinical Theme Active

02

Data Ingestion & Deduplication

Original dataset shape: (303, 14)
Found 1 duplicate rows. Dropping duplicates...
New dataset shape: (302, 14)
Total missing values: 0

03

Preprocessing Pipeline Setup

We split features conceptually and design a leak-free ColumnTransformer. Categorical variables are One-Hot Encoded, dropping the first category to avoid dummy variable multicollinearity (dummy trap). Continuous variables are scaled and skew-corrected.

Train Shape: (241, 13) | Test Shape: (61, 13)
ColumnTransformer constructed successfully.

04

Multicollinearity Audit (VIF)

We compute the Variance Inflation Factor (VIF) to identify multicollinearity among continuous variables. High collinearity (VIF > 5.0) can destabilize linear models like SVM and Logistic Regression.

VIF Audit:
    Feature    VIF
0       age  1.293
1  trestbps  1.108
2      chol  1.039
3   thalach  1.374
4   oldpeak  1.236

05

Model Training & Hyperparameter Tuning

We train 5 classifiers (Decision Tree, Random Forest, KNN, SVM, and XGBoost) using Scikit-Learn pipelines and GridSearchCV. The search is optimized for Accuracy on a Stratified 5-Fold Cross Validation.

Model: Decision Tree
  Best GridSearchCV CV Accuracy: 0.751
  Best Params: {'classifier__criterion': 'entropy', 'classifier__max_depth': 3, 'classifier__min_samples_split': 2}

Model: Random Forest
  Best GridSearchCV CV Accuracy: 0.822
  Best Params: {'classifier__max_depth': 5, 'classifier__min_samples_split': 2, 'classifier__n_estimators': 100}

Model: KNN
  Best GridSearchCV CV Accuracy: 0.826
  Best Params: {'classifier__n_neighbors': 13, 'classifier__p': 1, 'classifier__weights': 'uniform'}

Model: SVM
  Best GridSearchCV CV Accuracy: 0.830
  Best Params: {'classifier__C': 0.5, 'classifier__gamma': 'auto', 'classifier__kernel': 'rbf'}

Model: XGBoost
  Best GridSearchCV CV Accuracy: 0.805
  Best Params: {'classifier__learning_rate': 0.05, 'classifier__max_depth': 3, 'classifier__n_estimators': 50}

06

Advanced Ensembling (Stacking & Voting)

Ensemble pipelines fitted successfully.

07

Model Evaluation & Scores Comparison

	Accuracy	Precision (Class 1)	Recall (Class 1)	F1-Score (Class 1)	ROC-AUC	PR-AUC
KNN	0.869000	0.857000	0.909000	0.882000	0.903000	0.883000
Voting Classifier	0.869000	0.857000	0.909000	0.882000	0.909000	0.889000
SVM	0.852000	0.853000	0.879000	0.866000	0.912000	0.891000
Stacking Classifier	0.852000	0.853000	0.879000	0.866000	0.907000	0.891000
Random Forest	0.836000	0.848000	0.848000	0.848000	0.895000	0.878000
XGBoost	0.787000	0.794000	0.818000	0.806000	0.870000	0.854000
Decision Tree	0.738000	0.743000	0.788000	0.765000	0.820000	0.783000

ROC Curves and Precision-Recall Curves comparison across all models.

Confusion Matrix of our top-performing classifier.

08

Permutation Feature Importance